646 37,, 2.,, 1 Fig. 1 [1] (F,. P ω θ) The definition of disparity (F is the fixation point with zero disparity. The absolute disparity of P is ω θ.)

Size: px

Start display at page:

Download "646 37,, 2.,, 1 Fig. 1 [1] (F,. P ω θ) The definition of disparity (F is the fixation point with zero disparity. The absolute disparity of P is ω θ.)"

耻庆顾
5 years ago
Views:

1 37 6 Vol. 37, No ACTA AUTOMATICA SINICA June, ,.,,,,,, V1, V2, V3, MT, MST, IPS V4, IT. DOI,,,, /SP.J Disparity Computation in the Visual Cortices KONG Qing-Qun 1 MING Yan-Sheng 1 HU Zhan-Yi 1 Abstract The correspondence problem is essential for recovering the 3D shape of object from its images, however, this problem has not been solved satisfactorily in the computer vision field. Although numerous mathematical and engineering methods were reported in the literature, they were unable to perform as the human stereo vision does in terms of robustness. Therefore, a physiology based computational model is solicited, and this in turn requires some knowledge about biological stereo vision. To this end, we reviewed the relevant progresses in physiology about stereo depth perception, including the low level areas V1, V2, V3, dorsal areas MT, MST, IPS, and ventral areas V4, IT. In addition to these physiological studies, we also introduced the well known disparity energy model and its generalized model. This article could be of reference to computer vision researchers. Key words Stereo depth perception, low level visual areas, dorsal areas, ventral areas, disparity energy model (Stereopsis),, 20,,,,,,,,,,?,, Manuscript received September 3, 2010; accepted February 25, 2011 ( , ) Supported by National Natural Science Foundation of China ( , ) National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing ,. 1, F,,, P ω θ. 1,,. ( ),,.,,. Julesz (Random dot stereograms, RDS), 2,, Julesz RDS, Julesz. :,,,.,,

2 646 37,, 2.,, 1 Fig. 1 [1] (F,. P ω θ) The definition of disparity (F is the fixation point with zero disparity. The absolute disparity of P is ω θ.) 3, V1 V3, V4 IT (Inferior temporal cortex), V1 V2, MTA (Middle temporal area), MST (Middle superior temporal area) IPS (Intraparietal sulcus).. :,,. V1,,,,,.,,,. Fig. 2 2 The random dot stereogram,?,, V1 3 Fig. 3 The visual pathway V1 ( ),,

6 : 647 1.1.1 V1,,,, Barlow [2],, Hubel V1 [3 4],., Gabor [5], 4,,,,,,,. Fig. 4 4 Gabor The Gabor function, approximating simple cell RF., V1 [6].

5 Windowed grating stimuli (The rectangle represents the location of the receptive field.

3 6 : V1,,,, Barlow [2],, Hubel V1 [3 4],., Gabor [5], 4,,,,,,,. Fig. 4 4 Gabor The Gabor function, approximating simple cell RF., V1 [6]. ( 5), V1 [7] ; (RDS, ards), V1 ards [8], V1,. V1, Nienborg [9 10] V1, ( / ), V1,, 5 [7] ( (a) ; (b) (a), ) Fig. 5 Windowed grating stimuli (The rectangle represents the location of the receptive field. In (a), the corresponding parts of the stimuli overlie the receptive field; but in (b), the noncorresponding parts within the receptive field are identical to (a).) V1 (0 ),. Sanada Ohzawa [11] % (Frontoparallel plane),,, V1,, V1,, V [2], 1990, Ohzawa [12], 6., S, C X. 6,, 4., 4, 180, 90.

4 648 37,,, 7. V1. ards RDS,,,, Haefner, [13]. f R (x) = 1 ( ) exp x2 R cos(2πfx 2π 2σ 2 R + φ + φ) x R = x x f L (x) f R (x) x, φ.,., [14]., 8, C1, C2 : 8 [13] Fig. 8 The generalized energy model 6 [5] Fig. 6 Schematic representation of the disparity Fig. 7 energy model 7 [1] The disparity tuning curve,,,, f L (x) = 1 2π exp ( x2 2σ 2 ) cos(2πfx + φ),, (f 2 /f 1 = 2), x,, ϕ i = x f [15] i, ϕ 2 / ϕ 1 = 2, 9 ( )., 180,, 9 ( ) 1. V1, 42 %., 10. Haefner,,, C1, C2,, ards, 10,, 180., ards., (Extrastriate cortex (V1 )) [16].

6 : 649 0. 12 Shift-ratio. V2 Shift-ratio 0 1, V1 V2, V1 9 (a) (b) [13] Fig.

10 10 V1 ( [13] ) A sample neuron s response in V1 (The neuron concentrates its response near the physical disparity line),

11 [17] The response of a neuron selective for relative disparity to stimuli with varying central and surround Fig.

5 6 : Shift-ratio. V2 Shift-ratio 0 1, V1 V2, V1 9 (a) (b) [13] Fig. 9 The response of the disparity energy model with a pure position shift (a) or phase shift (b) Fig V1 ( [13] ) A sample neuron s response in V1 (The neuron concentrates its response near the physical disparity line), 20,, V1 V1 1.2 V2 V2 V3 ( 1.3 ), V1, V1, V2 V1., V2 [17]. 11.,.,,, Thomas (Shift-ratio) [17], Shift-ratio 1, 11 Fig. 11 [17] The response of a neuron selective for relative disparity to stimuli with varying central and surround Fig. 12 disparities 12 V1, V2 [17] The shift ratios were compared for neurons in V1 (a) and V2 (b), V2 (Cyclopean edge) [18]. Qiu V2 (Gestalt factors), (Borderownership coding) [19]., Bredfeldt RDS [20], V2. V2

6 (Figure-ground segmentation), V1, V2 [21]. (Choice probability, CP) [22]. RDS, RDS RDS,,, V2 Nienborg V2, V1. (Perceptual strategy) [23]. V1 V2?. V2, V2 V2, V2 Nienborg,, [24]. V1. V2 V1, V2 V1, V1, V2 (Optical imaging) V2 (Thick stripes) [25], V1 V2 V1,., MT (Middle temporal area) ( 2.1 ), V2, V2 V1,, V2 1.3 V3 V3, Adams, 61 % V3, V3 [26]., V3 [27]. V4, MT V3 V MT MT, V1, V2, V3, V1 [28]. V1, MT,,, MT. MT [29], V1, MT [30],. [31], MT,, MT. Nguyenkim RDS (72/97), [32].. MT, - (Center-surround RDS) MT [33 34]., MT. Uka [35],, MT, MT, MT. (Microstimulation) [36 37], MT, : 1) ( ); 2) ( ).,,, MT Shift-ratio ( 1.2 ), MT (Anti-correlated pattern), Dodd [38] CP

6 : 651 [39], MT, MT, 2.2 MST MST (Middle superior temporal area) MT, MT. MST ( ) (Far neurons) (Near neurons) [40], MST (Coarser stereopsis). MST Takemura (Vergence response), MST [41].

7 6 : 651 [39], MT, MT, 2.2 MST MST (Middle superior temporal area) MT, MT. MST ( ) (Far neurons) (Near neurons) [40], MST (Coarser stereopsis). MST Takemura (Vergence response), MST [41]., MST 4, 13,,,.,,, ards [42], RDS,,,, MST Ilg [43], MST,, (Post-saccade),. MST., MST, MST [43]. :,,,,, MST,,, MST, MST IPS (Intraparietal sulcus) ( 2.3 ). 13 Fig. 13 MST [41] The vergence eye movement could be approximated by the summed responses of different groups of MST neurons, MST MT,, MST. 2.3 IPS IPS,, - (Saccade). IPS,, AIP (Anterior intraparietal area), LIP (Lateral intraparietal area) MIP (Medial intraparietal area). Durand [44] (Functional magnetic resonance imaging, fmri), IPS, 2D, 1., AIP, MIP,,, Durand IPS [45],, DIPSA (The dorsal IPS anterior area) DIPSM (The dorsal IPS medial area) AIP, LIP,, VIPS/V7* (Ventral IPS area) CIP (Caudal intraparietal area),

8 IPS 2D [44] (, ) Table 1 Possible selectivities of areas in IPS for 3D shape, position, and 2D shape ( selectivity, nonselectivity) CIP 2D AIP LIP CIP PIP, MIP, AIP, Srivastava AIP [46], AIP. IPS (Fast) (Coarse) (Metric) ms, AIP (Preferred) (Non-preferred)., AIP, AIP. 14,, +., 0, AIP, AIP, AIP, AIP, AIP TEs ( 3.2 ),, TEs AIP [47 48]. Taira SFS (Solid figure stereogram) RDS [49], SFS, RDS CIP Durand [44] Taira , 45, 44 cm, Durand [47], CIP 14 AIP [46] Fig. 14 Metric selectivity of 3D shape in AIP, IPS -, AIP TEs V4 V4, V2, V3, MT, IT, V4 V1, V2 IT,, Hinkle V4, [50]., V1,, IT, V4 V1 IT. V1, V1, V2, V4, V4, 1/3 [51]. 15 Hinkle 1, 2, 3.,,,,

9 6 : 653, V4? Hedge [52] RDS, V4. V4 IT IPS ards. V4, IT? Tanabe V4 V4 RDS ards. IT, V4 V3. Kumano [55] ards,,, ards. Haefner 2SU [13] V1 [16]. V4 ards Fig Hinkle [51] Sketch map of Hinkle s experiment, V2, V4. Umeda V2, V4 [53]. Shift-ratio,. 16, Shift-ratio, Thomas V2 V1 [17]. V4 V2, Shift-ratio, V4., V4 V2. V2, V4 [53]., V4, V4,., RDS V4 ards [54]. ards, V4 V1, V2 V4, 16 V4 [53] Fig. 16 The comparison of selectivities for 3.2 TE relative disparity TE IT,, Jassen TE TEs (Superior temporal sulcus), TE (Lateral TE), TEs IPS, TE, TEs [47]. TEs,,, : Jassen TEs [56]., 17, 1 2

654 37 RDS, 3, 4.,. TEs, TEs,, 7..,,,, Fig.

,,, RDS, TEs,, Jassen TEs. TEs ards. TEs ards. Jassen IT [58]. TEs,., Yamane [59].

10 RDS, 3, 4.,. TEs, TEs,, 7..,,,, Fig TEs [57] This neuron is only responsive to curved surface, TEs? Jassen [57].,,, RDS, TEs,, Jassen TEs. TEs ards. TEs ards. Jassen IT [58]. TEs,., Yamane [59]. 18 TEs 50. Yamane TEs 18 Fig. 18 TEs 50 [59] Fifity 3D shapes causing an TEs neuron s highest response, TEs.. TEs,, TEs, IPS. 4 20,.,.,,.,,, :

11 6 : 655 1),,.,,?,,, 2), V1,,., Jassen, IT [58],,,.,,.,,. Grossberg Laminar cortical model (LCM) [60],,,,., LCM, 3),,,. Jassen [58],, IT???, ( ),,,, ( ). 4),,. Visual cognition,,,,?,,, V2, V3, V3,,,,. Bruce G. Cumming : /publications.asp Izumi Ohzawa : Guy A. Orban : katholieke/orban/guy-a-orban html Ralph D. Freeman : facprofiles/freeman.html References 1 Yu Lian-Qing. Stereo Matching with Cortical Disparity Detection Mechanisms [Ph. D. dissertation], Institute of Automation, Chinese Academy of Sciences, China, 2008 (. [ ],,, 2008) 2 Barlow H B, Blakemore C, Pettigrew J D. The neural mechanism of binocular depth discrimination. The Journal of Physiology, 1967, 193(2): Hubel D H, Wiesel T N. Receptive fields, binocular interaction and functional architecture in the cat s visual cortex. The Journal of Physiology, 1962, 160(1): Hubel D H, Wiesel T N. Receptive fields and functional architecture of monkey striate cortex. The Journal of Physiology, 1968, 195(1): Jones J P, Palmer L A. An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. Journal of Neurophysiology, 1987, 58(6): Cumming B G, Parker A J. Binocular neurons in V1 of awake monkeys are selective for absolute, not relative, disparity. The Journal of Neuroscience, 1999, 19(13): Cumming B G, Parker A J. Local disparity not perceived depth is signaled by binocular neurons in cortical area V1 of the macaque. The Journal of Neuroscience, 2000, 20(12): Cumming B G, Parker A J. Responses of primary visual cortical neurons to binocular disparity without depth perception. Nature, 1997, 389(6648): Nienborg H, Bridge H, Parker A J, Cumming B G. Receptive field size in V1 neurons limits acuity for perceiving disparity modulation. The Journal of Neuroscience, 2004, 24(9): Nienborg H, Bridge H, Parker A J, Cumming B G. Neuronal computation of disparity in V1 limits temporal resolution for detecting disparity modulation. The Journal of Neuroscience, 2004, 25(44):

12 Sanada T M, Ohzawa I. Encoding of three-dimensional surface slant in cat visual areas 17 and 18. Journal of Neurophysiology, 2006, 95(5): Ohzawa I, DeAngelis G C, Freeman R D. Stereoscopic depth discrimination in the visual cortex: neurons ideally suited as disparity detectors. Science, 1990, 249(4972): Haefner R M, Cumming B G. Adaptation to natural binocular disparities in primate V1 explained by a generalized energy model. Neuron, 2008, 57(1): Read J C, Cumming B G. Sensors for impossible stimuli may solve the stereo correspondence problem. Nature Neuroscience, 2007, 10(10): Fleet D J, Wagner H, Heeger D J. Neural encoding of binocular disparity: energy models, position shifts and phase shifts. Vision Research, 1996, 36(12): Tanabe S, Cumming B G. Mechanisms underlying the transformation of disparity signals from V1 to V2 in the macaque. The Journal of Neuroscience, 2008, 28(44): Thomas O M, Cumming B G, Parker A J. A specialization for relative disparity in V2. Nature Neuroscience, 2002, 5(5): Heydt V D R, Zhou H, Friedman H S. Representation of stereoscopic edges in monkey visual cortex. Vision Research, 2000, 40(15): Qiu F T, Heydt V D R. Figure and ground in the visual cortex: V2 combines stereoscopic cues with gestalt rules. Neuron, 2005, 47(1): Bredfeldt C B, Read J C A, Cumming B G. A quantitative explanation of responses to disparity-defined edges in macaque V2. Journal of Neurophysiology, 2009, 101(2): Nienborg H, Cumming B G. Macaque V2 neurons, but not V1 neurons, show choice-related activity. The Journal of Neuroscience, 2006, 26(37): Britten K H, Newsome W T, Shadlen M N, Celebrini S, Movshon J A. A relationship between behavioral choice and the visual responses of neurons in macaque MT. Visual Neuroscience, 1996, 13(1): Nienborg H, Cumming B G. Psychophysically measured task strategy for disparity discrimination is reflected in V2 neurons. Nature Neuroscience, 2007, 10(12): Nienborg H, Cumming B G. Decision-related activity in sensory neurons reflects more than a neuron s causal effect. Nature, 2009, 459(7243): Chen G, Lu H D, Roe A W. A map for horizontal disparity in monkey V2. Neuron, 2008, 58(3): Adams D L, Zeki S. Functional organization of macaque V3 for stereoscopic depth. Journal of Neurophysiology, 2001, 86(5): Georgieva S, Peeters R, Kolster H, Todd J T, Orban G A. The processing of three-dimensional shape from disparity in the human brain. The Journal of Neuroscience, 2009, 29(3): Born R T, Bradley D C, Structure and function of visual area MT. Annual Review of Neuroscience, 2005, 28(1): DeAngelis G C, Uka T. Coding of horizontal disparity and velocity by MT neurons in the alert macaque. Journal of Neurophysiology, 2003, 89(2): DeAngelis G C, Newsome W T. Organization of disparityselective neurons in macaque area MT. The Journal of Neuroscience, 1999, 19(4): Albright T D, Desimone R, Gross C G. Columnar organization of directionally selective cells in visual area MT of the macaque. Journal of Neurophysiology, 1984, 51(1): Nguyenkim J D, DeAngelis G C. Disparity-based coding of three-dimensional surface orientation by macaque middle temporal neurons. The Journal of Neuroscience, 2003, 23(18): Uka T, DeAngelis G C. MT neurons do not signal relative disparity. Journal of Vision, 2002, 2(7): Neri P, Bridge H, Heeger D J. Stereoscopic processing of absolute and relative disparity in human visual cortex. Journal of Neurophysiology, 2004, 92(3): Uka T, DeAngelis G C. Contribution of middle temporal area to coarse depth discrimination: comparison of neuronal and psychophysical sensitivity. The Journal of Neuroscience, 2003, 23(8): Uka T, DeAngelis G C. Linking neural representation to function in stereoscopic depth perception: roles of the middle temporal area in coarse versus fine disparity discrimination. The Journal of Neuroscience, 2006, 26(25): DeAngelis G C, Cumming B G, Newsome W T. Cortical area MT and the perception of stereoscopic depth. Nature, 1998, 394(6694): Dodd J V, Krug K, Cumming B G, Parker A J. Perceptually bistable three-dimensional figures evoke high choice probabilities in cortical area MT. The Journal of Neuroscience, 2001, 21(13): Krug K, Cumming B C, Parker A J. Comparing perceptual signals of single V5/MT neurons in two binocular depth tasks. Journal of Neurophysiology, 2004, 92(3): Roy J P, Komatsu H, Wurtz R H. Disparity sensitivity of neurons in monkey extrastriate area MST. The Journal of Neuroscience, 1992, 12(7): Takemura A, Inoue Y, Kawano K, Quaia C, Miles F A. Single-unit activity in cortical area MST associated with disparity-vergence eye movements: evidence for population coding. Journal of Neurophysiology, 2001, 85(5): Masson G S, Busettini C, Miles F A. Vergence eye movements in response to binocular disparity without depth perception. Nature, 1997, 389(6648): Ilg U J, Schumann S. Primate area MST-l is involved in the generation of goal-directed eye and hand movements. Journal of Neurophysiology, 2007, 97(1):

6 : 657 44 Durand J B, Nelissen K, Joly O, Wardak C, Todd J T, Norman J F, Janssen P, Vanduffel W, Orban G A. Anterior regions of monkey parietal cortex process visual 3D shape.

Neuroimage, 2009, 46(4): 1114 1126 46 Srivastava S, Orban G A, De Maziere P A, Janssen P.

The Journal of Neuroscience, 2009, 29(34): 10613 10626 47 Janssen P, Vogels R, Orban G A. Selectivity for 3D shape that reveals distinct areas within macaque inferior temporal cortex.

13 6 : Durand J B, Nelissen K, Joly O, Wardak C, Todd J T, Norman J F, Janssen P, Vanduffel W, Orban G A. Anterior regions of monkey parietal cortex process visual 3D shape. Neuron, 2007, 55(3): Durand J B, Peeters R, Norman J F, Todd J T, Orban G A. Parietal regions processing visual 3D shape extracted from disparity. Neuroimage, 2009, 46(4): Srivastava S, Orban G A, De Maziere P A, Janssen P. A distinct representation of three-dimensional shape in macaque anterior intraparietal area: fast, metric, and coarse. The Journal of Neuroscience, 2009, 29(34): Janssen P, Vogels R, Orban G A. Selectivity for 3D shape that reveals distinct areas within macaque inferior temporal cortex. Science, 2000, 288(5473): Janssen P, Vogels R, Orban G A. Macaque inferior temporal neurons are selective for disparity-defined three-dimensional shapes. Proceedings of the National Academy of Sciences of the United States of America, 1999, 96(14): Taira M, Tsutsui K I, Jiang M, Yara K, Sakata H. Parietal neurons represent surface orientation from the gradient of binocular disparity. Journal of Neurophysiology, 2000, 83(5): Hinkle D A, Connor C E. Quantitative characterization of disparity tuning in ventral pathway area V4. Journal of Neurophysiology, 2005, 94(4): Hinkle D A, Connor C E. Three-dimensional orientation tuning in macaque area V4. Nature Neuroscience, 2002, 5(7): Hegde J, Van Essen D C. Role of primate visual area V4 in the processing of 3-D shape characteristics defined by disparity. Journal of Neurophysiology, 2005, 94(4): Umeda K, Tanabe S, Fujita I. Representation of stereoscopic depth based on relative disparity in macaque area V4. Journal of Neurophysiology, 2007, 98(1): Tanabe S, Umeda K, Fujita I. Rejection of false matches for binocular correspondence in macaque visual cortical area V4. The Journal of Neuroscience, 2004, 24(37): Kumano H, Tanabe S, Fujita I. Spatial frequency integration for binocular correspondence in macaque area V4. Journal of Neurophysiology, 2008, 99(1): Janssen P, Vogels R, Orban G A. Three-dimensional shape coding in inferior temporal cortex. Neuron, 2000, 27(2): Janssen P, Vogels R, Liu Y, Orban G A. Macaque inferior temporal neurons are selective for three-dimensional boundaries and surfaces. The Journal of Neuroscience, 2001, 21(23): Janssen P, Vogels R, Liu Y, Orban G A. At least at the level of inferior temporal cortex, the stereo correspondence problem is solved. Neuron, 2003, 37(4): Yamane Y, Carlson E T, Bowman K C, Wang Z H, Connor C E. A neural code for three-dimensional object shape in macaque inferotemporal cortex. Nature Neuroscience, 2008, 11(11): Grossberg S, Howe P D. A laminar cortical model of stereopsis and three-dimensional surface perception. Vision Research, 2003, 43(7): qqkong@nlpr.ia.ac.cn (KONG Qing-Qun Ph. D. candidate at the Institute of Automation, Chinese Academy of Sciences. She received her bachelor degree from Ocean University of China in Her research interest covers computational theories and models based on human vision mechanism. Corresponding author of this paper.) mingyansh@hotmail.com (MING Yan-Sheng Ph. D. candidate at the College of Engineering and Computer Science, Australian National University. He received his bachelor degree in the Department of Control Science and Engineering, Zhejiang University in He received his master degree in pattern recognition and machine learning from the Institute of Automation, Chinese Acaderny of Sciences in His research interest covers computational theories and models based on human vision mechanism.). huzy@nlpr.ia.ac.cn (HU Zhan-Yi Professor at the Institute of Automation, Chinese Academy of Science. His research interest covers camera calibration, 3D reconstruction, and vision guided robot navigation.)

2 XX,,,,.Julesz (random dot stereograms, RDS), 2,,..Julesz RDS,.Julesz. :,.,,.,,,, 2..,,.,,.,,.,,...,,,.,.,,.,,. MT MST IPS V1 V2 V3 V4 IT Fig. 2 2 Th

2 XX,,,,.Julesz (random dot stereograms, RDS), 2,,..Julesz RDS,.Julesz. :,.,,.,,,, 2..,,.,,.,,.,,...,,,.,.,,.,,. MT MST IPS V1 V2 V3 V4 IT Fig. 2 2 Th XX X Vol. XX, No. X 200X X ACTA AUTOMATICA SINICA Month, 200X 1 1 1,.,,.,,.,,.,, V1,V2,V3, MT,MST,IPS V4,IT.,.. DOI,,,, 10.3724/SP.J.1004.2008.xxxxx Disparity Computation in the Visual Cortices Abstract