In a short treatise on visual perception, Ernst Mach wrote that, "we do not see optical images in an optical space, but we perceive the bodies round about us in their many and sensuous qualities." To put it in more modern parlance, "A central puzzle in vision is how the brain infers properties of the world ('bodies round about us') from uncertain and ambiguous sensory data ('optical images in an optical space').
While understanding the processes that lead to our conscious perception of the world is important and interesting, perhaps more fundamental to human behavior in the natural world are the processes by which the brain uses visual information to guide motor behavior. These processes occur "under the radar" of consciousness and in some cases appear to be distinct from those that determine our conscious percepts – neuropsychological dissociations between vision for perception and vision for action abound in the literature.
The brain has a multitude of sensory cues about the world to make perceptual inferences and to guide motor behavior. Vision itself contains a number of qualitatively different cues that are coded by at least partially distinct neural mechanisms (stereoscopic disparities, motion, shading, texture, contour, shadows, etc.). Other sensory modalities provide independent sources of information as well. Touch can provide information about an object's shape, orientation and position in space, kinesthetic cues provide information about limb and body movements, audition provides information about an object's position, motion and material makeup.
One of the striking findings in visual science from the past several decades is that the brain only retains a very small amount of information about visual scenes from one time to the next; for example, across saccadic gaze shifts. This has led some researchers to study the nature of the capacity limits of visual working memory and how that information is represented in the brain. We develop and test information theoretic computational models that emphasize both the limits on and flexibility of memory encoding. We are further extending "classic" visual working memory experiments, which use kinds of explicit visual memory tasks that we rarely perform in everyday life, to study how observers use visual working memory when performing natural tasks. In these experiments, we study visual working memory as it is used to guide hand movements when performing naturalistic tasks in virtual reality. This allows us to study how the demands of natural tasks shape how the brain represents information in working memory.
The world is a highly structured place with a tremendous amount of statistical structure. Not all objects or events are equally likely to occur – in fact, most physically realizable objects and events almost never occur in our world. The brain uses knowledge of this statistical structure improve performance in a number of ways. It uses statistical knowledge to disambiguate uncertain and ambiguous sensory information, to efficiently code information in working memory and to better perform sensorimotor tasks. Recently, a number of researchers including ourselves have shown that the visual system, and sensory systems in general, do not use static knowledge of environmental statistics, but rather adapts its internal models as the statistics of one's local environment changes (e.g. as you go from your office to the woods). We study how the brain adapts its computations and representations to the changing statistics of visual scenes using both computational models and psychophysical experiments.
Computational modeling work focuses on constructing Bayesian theories of both the information in images that is, in theory, available to the brain (ideal observer models) for performing tasks and similar theories of how the brain uses the information to make perceptual inferences about objects in scenes (Bayesian observer models) and to guide motor actions (like reaching for objects). Using a common mathematical framework for modeling how a task can be performed in theory (theories of competence) and how humans actually perform the tasks (theories of performance) allows us to determine what aspects of human performance are determined by the structure of available visual information and of task demands and what are due to limitations in how the brain represents and does computations on visual information.