Color constancy refers to the stability of object color appearance despite changes in illumination. Color constancy is an important problem because it embodies an ambiguity that is at the core of all perceptual problems: multiple physical configurations can produce the same image. In color constancy, the ambiguity arises because we can trade off illuminant and surface spectra while keeping the image constant.

In this talk, I will review experiments conducted in my laboratory that measure human color constancy under nearly natural viewing conditions. The first set of experiments provide baseline measurments. These experiments show that under rich viewing conditions, color constancy is a robust and regular effect that may be studied quantitatively. The second set of experiments systematically manipulates the richness of the viewing conditions. These experiments rule out the possibility that constancy results entirely from the action of simple low-level processes such as simultaneous color contrast or chromatic adaptation to the spatial mean of the image.

Color appearance depends on many aspects of vision that follow light absorption by receptors. Perception of natural scenes, which are composed of meaningful objects that form a complex mosaic of chromaticities, involves perceptual processes that are seldom studied in simpler laboratory experiments. Two such processes are considered here: (i) classification of objects according to a common illuminant and (ii) color memory.

Classification of objects by illumination is a different problem than color constancy. In color constancy, the aim is a color representation of an object that does not depend on the particular illuminating light. For example, an approach to color constancy is to represent object colors in a way that takes account of ("discounts") the inferred illumination. The classification-by-illumination problem, on the other hand, considers how the parts of a complex stimulus are separated into those regions sharing the same illuminant. Experiments show that minimal changes in the spatial configuration of ten distinct patches of light can alter the illuminant classification, and result in a surprising change in perceived brightness. Lower inferred illumination, without an actual change of light entering the eye, makes a fixed light appear dimmer. Similar results are found with color: a minimal reconfiguration of a fixed set of chromatic stimuli causes color appearance to shift in the direction of the inferred illumination.

The utility of object colors depends on color memory. Consider finding your navy-blue coat in a crowded coat closet, recognizing healthy from poorly watered grass, or observing a symptom of disease characterized by subtle changes in the color of skin or retina. How is the color that one learns encoded and stored in memory? Experiments show that (a) the remembered color of a patch is less affected by the illumination during learning when the color is learned with many different chromaticities also in view, compared to learning with a uniform background (even a background at the chromaticity of the illuminant); and (b) a color recalled after 10 minutes is less affected by the illuminant during learning than a color recalled after 10 seconds. These results suggest that colors viewed within complex scenes and held in memory for many minutes, as in natural vision, tend toward object colors more than would be expected from measurements of simultaneous color constancy or successive color matching.

Laboratory measurements of colour constancy are typically made for 2D surfaces with spatially uniform surface reflectance properties under uniform direct illumination. Colour appearance and colour constancy for simple scenes with these characteristics may largely be explained by low-level sensory mechanisms such as cone adaptation and local contrast computations.

Colour constancy of 3D objects is more difficult to quantify (but I will describe some attempts to do so). 3D objects in real scenes typically have large surface chromaticity and luminance variations, due to inherent surface reflectance variations, indirect illumination from other objects (mutual illumination), multiple light sources, and 3D surface shading. Mutual illumination, in particular, may have strong effects on surface chromaticity and luminance. Mutual illumination may also provide a cue to surface reflectance, and therefore has been proposed as a factor that may enhance colour constancy in the real world.

The Chromatic Mach Card demonstrates an intrinsic link between the perception of surface colour, 3D shape, and mutual illumination. The Chromatic Mach Card is a concave folded card with one side made of red paper and the other of white paper. The light reflected from the red side casts a pinkish glow on the white side, via mutual illumination. The perceived colour of the white side dramatically changes from white to pink when the perceived shape of the card flips from concave to convex. The effect cannot be explained in the same way as the traditional achromatic Mach Card, but instead is consistent with a Bayesian analysis of generic properties in which the visual system seeks image interpretations which are robust over varying colour and position of the light source. For a convex geometry in which the red card does not face the white one, red and pink surface colours are less accidental than red and white surface colours. For the concave geometry, the visual system appears to use its knowledge of the relationship of shape and mutual illumination to discount the reddish indirect light and to recover correctly the surface colour of the white paper.

Thus: (1) the human visual system incorporates knowledge of mutual illumination to recover surface colour; (2) perceived surface colour is not a property of an object that can be dissociated from its 3D shape; and (3) to interpret physically ambiguous images, the visual system appears to accept the least accidental set of object properties, assuming unknown variations in genetic environmental variables.

There are a number of visual cues that can, in principle, provide information about the shape, location, and material surface properties of each small surface patch in a given scene. It is well known, for example, that multiple depth cues influence human perception of the locations of objects relative to the observer. When several cues are simultaneously available, for a single location, the visual system may attempt to combine them. I'll discuss three key issues relevant to the experimental analysis of visual cue combination in human vision, and review recent psychophysical and computational studies of human cue combination in light of these issues. The discussion and review are organized as the development of a model of the cue combination process termed "modified weak fusion" (MWF). this model was originally developed as as model of the depth and shape cue combination process in human vision. I'll describe how it might be extended to surface color perception by considering illuminant estimation as a cue combination process.

While the MWF model is motivated by normative considerations, it is primarily intended to guide experimental analysis of cue combination in human vision. I'll describe experimental methods that permit us to analyze cue combination in novel ways. In particular, these methods allow us to investigate the key issues mentioned above. I'll summarize recent experimental tests of the MWF framework for depth cue combination that use these methods, and describe work in progress relevant to surface color perception.

Theories of binocular vision fall into two broad classes. Image based theories address the "correspondence" problem as the primary challenge, asking two questions: What image feature in one eye is matched to the same feature in the other, and how is this matching implemented? Scene based views argue that binocular vision should be considered more generally, asking what surfaces in the world are the most likely to have given rise to the image data.

In this talk, I outline a sample of four topics, assuming a scene based view of binocular vision. In particular I will (1) Examine binocular vision in terms of the generic view principle (2) Outline Da Vinci stereopsis (based on unpaired image regions) in terms of occlusion relationships (3) Extend the treatment of unpaired points, outlining a taxonomy of surface interpretations of such unpaired image regions in terms of border ownership assignments (4) Further question the necessity of binocular matching, specifically the polarity of edge assignments, for quantitative stereopsis.

(with Duane G. Albrecht, Robert A. Frazor, & Alison M. Crane)

An ultimate goal of vision science is to understand the neural information processing of natural visual stimuli and to predict behavioral performance in natural visual tasks. We are attempting to make some progress toward this goal by quantitatively analyzing the responses of single neurons in primary visual cortex and by comparing the responses of populations of single neurons with behavioral performance. Our approach consists of three major steps. First, the responses of individual cortical neurons are measured for sine wave grating stimuli varied along a number of stimulus dimensions: spatial frequency, orientation, phase, contrast, temporal frequency, direction of motion. Second, the measured response means and standard deviations are fitted with a descriptive functional model (one for each neuron measured), which can be used to determine the response of the neuron to any sine wave stimulus, or to estimate the response of the neuron to any arbitrary stimulus (e.g., a complex natural scene). Third, the descriptive model for each measured neuron is combined with a decision/pooling rule to determine the performance of the neural population in some task. The performance of the neural population is then compared with behavioral performance in the same task. The validity of this approach depends upon several factors: the accuracy of the functional descriptive model, the degree of statistical independence of the single neuron responses, the form of the decision/pooling rule. After discussing these factors, I will describe results for contrast discrimination, spatial frequency discrimination and feature identification in complex natural images.

The notion of the temporal lobe being involved in object recognition—as it emerged from clinical and lesion studies—has received strong support from previous electrophysiological experiments in monkeys. The results of these experiments showed that neurons in the inferior temporal cortex (IT) respond to a variety of complex two-dimensional patterns, including figures of animate objects, such as faces, hands, and body parts. To better understand the role of this area in object recognition, we set out to determine whether the configurational selectivity found for IT neurons is specific for faces or body parts, or whether it can be generated for any novel object as a result of extensive training.

Monkeys were trained to become "experts" at identifying exemplars of novel, computer-generated object classes. Critically, these objects had never been experienced by the monkeys, nor did they possess any inherent biological relevance. Nonetheless, after training, the animals learned to discriminate individual objects from a set of highly similar distractors, a task not unlike that of identifying a specific face or a particular bird species. Because all of the objects used in testing were composed of the same basic parts, good performance in this task had most likely to rely upon using holistic configurational information, and upon the detection of subtle shape differences.

Physiological recordings from individual neurons in IT revealed a subpopulation of cells that were activated selectively by views of these previously unfamiliar objects. Many neurons fired selectively for a small set of views of spheroidal or wire objects that the monkey had learned to recognize from all viewpoints. The cells were most active when the target was presented from one particular view, and their activity declined as the object was rotated in depth. Remarkably, these cells could not be consistently activated by any other tested object, including numerous visually similar distractors and views of the target more than about 45 deg away from the preferred view. Attempts to simplify the objects lead, in most cases, to a significant reduction of the neurons' responses, a finding suggesting that some neurons in IT show the type of specificity to complex configurations, which was previously described for faces or other animate objects. Testing IT neurons with stimuli that can be perceived more than one way showed that these cells discharge for their effective stimulus only when this stimulus is perceived. Phenomenal suppression of the preferred pattern caused almost invariably a profound suppression of the cells' activity. IT thus seems to be very closely related not only to shape representation but also to the conscious perception of a visual object.