Plenoptic Sampling Jin-Xiang Chai Xin Tong Shing-Chow Chan y Heung-Yeung Shum z Microsoft Research, China Abstract This paper studies the problem of p

Transcription

1 Plenopic Sampling Jin-Xiang Chai Xin Tong Shing-Chow Chan y Heung-Yeung Shum z Microsof Research, China Absrac This paper sudies he problem of plenopic sampling in imagebased rendering (IBR). From a specral analysis of ligh field signals and using he sampling heorem, we mahemaically derie he analyical funcions o deermine he minimum sampling rae for ligh field rendering. The specral suppor of a ligh field signal is bounded by he minimum and maximum dephs only, no maer how complicaed he specral suppor migh be because of deph ariaions in he scene. The minimum sampling rae for ligh field rendering is obained by compacing he replicas of he specral suppor of he sampled ligh field wihin he smalles ineral. Gien he minimum and maximum dephs, a reconsrucion filer wih an opimal and consan deph can be designed o achiee ani-aliased ligh field rendering. Plenopic sampling goes beyond he minimum number of images needed for ani-aliased ligh field rendering. More significanly, i uilizes he scene deph informaion o deermine he minimum sampling cure in he join image and geomery space. The minimum sampling cure quaniaiely describes he relaionship among hree key elemens in IBR sysems: scene complexiy (geomerical and exural informaion), he number of image samples, and he oupu resoluion. Therefore, plenopic sampling bridges he gap beween image-based rendering and radiional geomerybased rendering. Experimenal resuls demonsrae he effecieness of our approach. Keywords: sampling, plenopic sampling, specral analysis, plenopic funcions, image-based rendering. 1 Inroducion Image-based modeling and rendering echniques hae recenly receied much aenion as a powerful alernaie o radiional geomery-based echniques for image synhesis. Insead of geomerical primiies, a collecion of sample images are used o render noel iews. Preious work on image-based rendering (IBR) reeals a coninuum of image-based represenaions [15, 14] based on he radeoff beween how many inpu images are needed and how much is known abou he scene geomery. A one end, radiional exure mapping relies on ery accurae geomerical models bu only a few images. In an image-based rendering sysem wih deph maps, such as 3D warping [18], iew Currenly a Carnegie Mellon Uniersiy. jchai@cs.cmu.edu y Visiing from Uniersiy of Hong Kong. scchan@eee.hku.hk z fxong,hshumg@microsof.com inerpolaion [7], iew morphing [21] and layered-deph images (LDI) [22], LDI ree [6], ec., he model consiss of a se of images of a scene and heir associaed deph maps. When deph is aailable for eery poin in an image, he image can be rendered from any nearby poin of iew by projecing he pixels of he image o heir proper 3D locaions and re-projecing hem ono a new picure. A he oher end, ligh field rendering uses many images bu does no require any geomerical informaion. Ligh field rendering [16] generaes a new image of a scene by appropriaely filering and inerpolaing a pre-acquired se of samples. Lumigraph [10] is similar o ligh field rendering bu i applies approximaed geomery o compensae for non-uniform sampling in order o improe rendering performance. Unlike ligh field and Lumigraph where cameras are placed on a wo-dimensional manifold, Concenric Mosaics sysem [23] reduces he amoun of daa by only capuring a sequence of images along a circular pah. Ligh field rendering, howeer, ypically relies on oersampling o couner undesirable aliasing effecs in oupu display. Oersampling means more inensie daa acquisiion, more sorage, and more redundancy. To dae, lile research has been done on deermining he lower bound or he minimum number of samples needed for ligh field rendering. Sampling analysis in IBR is a difficul problem because i inoles he complex relaionship among hree elemens: he deph and exure informaion of he scene, he number of sample images, and he rendering resoluion. The opic of prefilering a ligh field has been explored in [16]. Similar filering process has been preiously discussed by Halle [11] in he conex of Holographic sereograms. A parameerizaion for more uniform sampling [4] has also been proposed. From an iniially undersampled Lumigraph, new iews can be adapiely acquired if he rendering qualiy can be improed [20]. An opposie approach is o sar wih an oersampled ligh field, and o cull an inpu iew if i can be prediced by is neighboring frames [12, 24]. Using a geomerical approach and wihou considering exural informaion of he scene, Lin and Shum [17] recenly sudied he number of samples needed in ligh field rendering wih consan deph assumpion and bilinear inerpolaion. Howeer, a mahemaical framework has no been fully deeloped for sudying he sampling problems in IBR. In his paper, we sudy plenopic sampling, or how many samples are needed for plenopic modeling [19, 1]. Plenopic sampling can be saed as: How many samples of he plenopic funcion (e.g., from a 4D ligh field) and how much geomerical and exural informaion are needed o generae a coninuous represenaion of he plenopic funcion? Specifically, our objecie in his paper is o ackle he following wo problems under plenopic sampling, wih and wihou geomerical informaion: Minimum sampling rae for ligh field rendering; Minimum sampling cure in join image and geomery space. We formulae he sampling analysis as a high dimensional signal processing problem. In our analysis, we assume Lamberian surfaces and uniform sampling geomery or laice for he ligh field.

2 f z(,) f z (, ) 0 (a) (b) 0 Figure 1: An illusraion of 2D ligh field or EPI: (a) a poin is obsered by wo cameras 0 and ; (b) wo lines are formed by sacking pixels capured along he camera pah. Each line has a uniform color because of Lamberian assumpion on objec surfaces. Raher han aemping o obain a closed-form general soluion o he 4D ligh field specral analysis, we only analyze he bounds of he specral suppor of he ligh field signals. A key analysis o be presened in his paper is ha he specral suppor of a ligh field signal is bounded by only he minimum and maximum dephs, irrespecie of how complicaed he specral suppor migh be because of deph ariaions in he scene. Gien he minimum and maximum dephs, a reconsrucion filer wih an opimal and consan deph can be designed o achiee ani-aliased ligh field rendering. The minimum sampling rae of ligh field rendering is obained by compacing he replicas of he specral suppor of he sampled ligh field wihin he smalles ineral wihou any oerlap. Using more deph informaion, plenopic sampling in he join image and geomery space allows us o grealy reduce he number of images needed. In fac, he relaionship beween he number of images and he geomerical informaion under a gien rendering resoluion can be quaniaiely described by a minimum sampling cure. This minimal sampling cure seres as he design principles for IBR sysems. Furhermore, i bridges he gap beween image-based rendering and radiional geomery-based rendering. Our approach is inspired by he work on moion compensaion filer in he area of digial ideo processing, in which deph informaion has been incorporaed ino he design of he opimal moion compensaion filer [25, 9]. In digial ideo processing, global consan deph and arbirary moion are considered for boh saic and dynamic scenes, whereas in our work, we analyze saic scenes wih an arbirary geomery and wih uniformly sampled camera seups. The remainder of his paper is organized as follows. In Secion 2, a specral analysis of 4D ligh field is inroduced and he bounds of is specral suppor are deermined. From hese bounds, he minimum sampling rae for ligh field rendering can be deried analyically. Plenopic sampling in he join image and geomery space is sudied in Secion 3. The minimum sampling cures are deduced wih accurae and approximaed dephs. Experimenal resuls are presened in Secion 4. Finally we conclude our paper in Secion 5. 2 Specral analysis of ligh field 2.1 Ligh field represenaion We begin by briefly reiewing he properies of ligh field represenaion. We will follow he noaions in he Lumigraph paper [10]. In he sandard wo-plane ray daabase parameerizaion, here is a camera plane, wih parameer (s ), and a focal plane, wih parameer (u ). Each ray in he parameerizaion is uniquely deermined by he quadruple (u s ). We refer he reader o Figure 2(a) of [10] for more deails. A wo dimensional subspace gien by fixed alues of s and resembles an image, whereas fixed alues of u and gie a hypoheical radiance funcion. Fixing and gies rise o an epipolar image, or EPI [3]. An example of a 2D ligh field or EPI is shown in Figure 1. Noe ha in our analysis we define (u ) in he local coordinaes of (s ), unlike in conenional ligh field where (u s ) are defined in a global coordinae sysem. Assume he sample inerals along s and direcions be s and, respeciely, he horizonal and erical dispariies beween wo grid cameras in he (s ) plane are deermined by k 1sf=z and k 2f=z, respeciely, where f denoes he focal lengh of he camera, z is he deph alue and (k 1s k 2) is he sample ineral beween wo grid poins (s ). Similarly, we assume ha he sample inerals along u and direcions be u and, respeciely. A pinhole camera model is adoped o capure he ligh field. Wha a camera sees is a blurred ersion of he plenopic funcion because of finie camera resoluion. A pixel alue is a weighed inegral of he illuminaion of he ligh arriing a he camera plane, or he conoluion of he plenopic funcion wih a low-pass filer. 2.2 A framework for ligh field reconsrucion Le l(u s ) represen he coninuous ligh field, p(u s ) he sampling paern in ligh field, r(u s ) he combined filering and inerpolaing low-pass filer, and i(u s ) he oupu image afer reconsrucion. Le L P R and I represen heir corresponding specra, respeciely. In he spaial domain, he ligh field reconsrucion can be compued as i(u s ) =r(u s ) [l(u s )p(u s )] (1) where represens he conoluion operaion. In he frequency domain, we hae I( u s )=R( u s )(L( u s ) P ( u s )) (2) The problem of ligh field reconsrucion is o find a reconsrucion filer r(u s ) for ani-aliased ligh field rendering, gien he sampled ligh field signals. 2.3 Specral suppor of ligh fields In his secion, we will inroduce he specral suppors of coninuous ligh field L( u s ) and sampled ligh field L( u s ) P ( u s ) Specral suppor of coninuous ligh field We assume ha he deph funcion of he scene is equal o z(u s ). As shown in Figure 1(a), he same 3D poin is obsered a 0 and in he local coordinae sysems of cameras 0 and

3 , respeciely. The dispariy beween he wo image coordinaes can be compued easily as ; 0 = f=z. Figure 1(b) shows an EPI image where each line represens he radiance obsered from differen cameras. For simpliciy of analysis, he BRDF model of a real scene is assumed o be Lamberian. Therefore, each line in Figure 1(b) has a uniform color. Therefore, he radiance receied a he camera posiion (s ) is gien by l(u s ) =l(u ; fs z(u s ) ; f 0 0) z(u s ) and is Fourier ransform is L( u s )= l(u s )e ;jt x dx ;1 ;1 ;1 e ;j(ss+) dsd (3) where x T =[u ] and T =[ u ]. Howeer, compuing he Fourier ransform (3) is ery complicaed, and we will no go ino he deails of is deriaion in his paper. Insead, we will analyze he bounds of he specral suppor of ligh fields. Also for simpliciy, i is assumed ha samples of he ligh field are aken oer he commonly used recangular sampling laice Specral suppor of sampled ligh field Using he recangular sampling laice, he sampled ligh field l s(u s ) is represened by l s(u s ) =l(u s ) X n 1 n 2 k 1 k 2 2 (u ; n 1u)( ; n 2)(s ; k 1s)( ; k 2) (4) and is Fourier ransform is L( u ; 2m 1 u L s( u s )= ; 2m2 s ; 2l1 s X m 1 m 2 l 1 l 2 2 ; 2l2 ) (5) The aboe equaion indicaes ha L s( u s ) consiss of replicas of L( u s ), shifed o he 4D grid poins (2m 1=u 2m 2= 2l 1=s 2l 2=) where m 1 m 2 l 1 l 2 2, and is he se of inegers. These shifed specra, or replicas, excep he original one a m 1 = m 2 = l 1 = l 2 =0, are called he alias componens. When L is no bandlimied ouside he Nyquis frequencies, some replicas will oerlap wih he ohers, creaing aliasing arifacs. In general, here are wo ways o comba aliasing effecs in oupu display when we render a noel image. Firs, we can increase he sampling rae. The higher he sampling rae, he less he aliasing effecs. Indeed, uniform oersampling has been consisenly employed in many IBR sysems o aoid undesirable aliasing effecs. Howeer, oersampling means more effor in daa acquisiion and requires more sorage. Though redundancy in he oersampled image daabase can be parially eliminaed by compression, excessie samples are always waseful. Second, ligh field signals can also be made bandlimied by filering wih an appropriae filer kernel. Similar filering has o be performed o remoe he oerlapping of alias componens during reconsrucion or rendering. The design of such a kernel is, howeer, relaed o he deph of he scene. Preious work on Lumigraph shows ha approximae deph correcion can significanly improe he inerpolaion resuls. The quesions are: is here an opimal filer? Gien he number of samples capured, how accuraely should he deph be recoered? Similarly, gien he deph informaion one can recoer, how many samples can be remoed from he original inpu? 2.4 Analysis of bounds in specral suppor A model of global consan deph Le us firs consider he simples scene model in which eery poin is a a consan deph (z 0). The firs frame is chosen as he reference frame, and l(u 0 0) denoes he 2D inensiy disribuion wihin he reference frame. The 4D Fourier ransform of he ligh field signal l(u s ) wih consan deph is 1 1 L( u s )= l(u 0 0)e ;j(uu+) dud ;1 ;1 1 1 e ;j( fz u+s)s 0 ds ;1 e ;j( fz 0 +) d ;1 =4 2 L 0 ( u )( f z 0 u + s)( f z 0 + ) where L 0 ( u ) is he 2D Fourier ransform of coninuous signal l(u 0 0) and () is he 1D Dirac dela funcion. To keep noaion, represenaions and illusraion simple, he following discussion will focus on he projecion of he suppor of L( u s ) ono he ( ) plane, which is denoed by L( ). Under he consan deph model, he specral suppor of he coninuous ligh field signal L( ) is defined by a line f=z 0 + =0, as shown in Figure 2(b). The specral suppor of he corresponding sampled ligh field signals is shown in Figure 2(c). Noe ha, due o sampling, replicas of L( ) appear a inerals 2m 2= and 2l 2= in he and direcions, respeciely. Figure 6(a) shows a consan deph scene (a1), is EPI image (a2), and he Fourier ransform of he EPI (a3). As expeced, he specral suppor is a sraigh line Spaially arying deph model Now i is sraighforward o obsere ha any scene wih a deph beween he minimum z min and he maximum z max will hae is coninuous specral suppor bounded in he frequency domain, by wo lines f=z min + =0and f=z max + =0. Figure 6(b3) shows he specral suppor when wo planes wih consan dephs are in he scene. Adding anoher iled plane in beween (Figure 6(c1)) resuls in no ariaions in he bounds of he specral suppor, een hough he resuling specral suppor (Figure 6(c3)) differs significanly from ha in Figure 6(c2). This is furher illusraed when a cured surface is insered in beween wo original planes, as shown in Figure 6(d1). Een hough he specral suppors differ significanly, Figures 6(b3), (c3) and (d3) all hae he same bounds. Anoher imporan obseraion is ha geomerical informaion can help o reduce he bounds of he specral suppor in he frequency domain. As will be illusraed in he following secion, he opimal reconsrucion filer is deermined precisely by he bounds of he specral suppor. And hese bounds are funcions of he minimum and maximum dephs of he scene. If some informaion on he scene geomery is known, we can decompose he scene geomery ino a collecion of consan deph models on a block-by-block basis. Each model will hae a much igher bound han he original model. How igh he bound is will depend on he accuracy 1 The ringing effec in he iciniy of he horizonal and erical axes is caused by conoling wih sin( )= because of he recangular image boundary.

4 Ω Ω 0 f 0 Ω Ω +Ω = 0 (a) (b) (c) 2π 2π Ω Figure 2: Specral suppor of ligh field signals wih consan deph: (a) a model of consan deph; (b) he specral suppor of coninuous ligh field signals; (c) he specral suppor of sampled ligh field signals. max 0 f 0 Ω Ω + Ω = 0 π f min Ω + Ω = 0 Ω min (a) f max Ω + Ω = 0 π (b) Figure 3: Specral suppor for ligh field signal wih spaially arying dephs: (a) a local consan deph model bounded by z min and z max is augmened wih anoher deph alue z 0; (b)specral suppor is now bounded by wo smaller regions, wih he inroducion of he new line of z 0. Ω Ω aliasing Ω Ω Ω Ω Ω Ω (a) (b) (c) (d) Figure 4: Three reconsrucion filers wih differen consan dephs: (a) infinie deph; (b) infinie deph (aliasing occurs); (c) maximum deph; (d) opimal deph a z c. Ω Ω K Ω Ω Ω K Ω Pmax Pmin l 4 l (a) (b) Figure 5: (a) The smalles ineral ha replicas can be packed wihou any oerlap is P maxp min, deermined by he highes frequency K. (b) A specral suppor decomposed ino muliple layers.

5 (a1) Scene image (a2) EPI (a3) Fourier ransform of EPI (b1)scene image (b2) EPI (b3) Fourier ransform of EPI (c1) Scene image (c2) EPI (c3) Fourier ransform of EPI (d1)scene image (d2) EPI (d3) Fourier ransform of EPI Figure 6: Specral suppor of a 2D ligh field: (a) a single plane; (b) wo planes; (c) a hird and iled plane in beween; (d) a cured surface in beween.

6 of he geomery. Figure 3 illusraes he reducion in bounds, from [z min z max] o max([z min z 0] [z 0 z max]), wih he inroducion of anoher layer A model wih runcaing windows Because of he lineariy of he Fourier ransform, he specral suppor of he EPI image for a scene wih wo consan planes will be wo sraigh lines. Howeer, his saemen is rue only if hese wo planes do no occlude each oher. For synheic enironmens, we can consruc such EPI images on differen layers by simply ignoring he occlusion. In pracice, we can represen a complicaed enironmen using a model wih runcaing windows. For example, we can approximae an enironmen using runcaed and piece-wise consan deph segmens. Specifically, suppose he deph can be pariioned as z() =z i for i <i+1 i=1 N d where 1 and Nd +1 are he smalles and larges of ineres respeciely. Then and L( ) = l( ) =l i( ; f=z i 0) if i <i+1 N X d i=1 exp(;j i + i+1 2 ( + z i=f)) 2sin( i+1; i ( 2 + z i=f)) L i(; z i=f) f =z i + N X d i=1 Q i( ) (6) where L i is he 1D Fourier ransform of l i. In (6), because he funcion sin x x decays fas, and L i(; z i=f) also decreases fas when j j grows, he specral suppor of Q i( ) will look like a narrow ellipse. Neerheless, because of high frequency leak, cu-off frequency should be used in he sampling analysis. An example of wo consan planes in an enironmen is shown in Figures 6(b1) (original image), 6(b2) (EPI) and 6(b3) (specral suppor). Noe ha he shape of each of he wo specral suppors, i.e., wo approximaed lines, is no significanly affeced by occlusion because he widh of each specral suppor is no oo large. 2.5 A reconsrucion filer using a consan deph Gien a consan deph, a reconsrucion filer can be designed. Figure 4 illusraes four differen designs of reconsrucion filers oriened o differen consan dephs. Aliasing occurs when replicas oerlap wih he reconsrucion filers in he frequency domain ( and ), as shown in Figure 4(a)(b)(d). Ani-aliased ligh field rendering can be achieed by applying he opimal filer as shown in Figure 4(c), where he opimal consan deph is defined as he inerse of aerage dispariy d c, i.e., d c = 1 1 =( + 1 )=2: z c z min z max Figure 7 shows he effec of applying reconsrucion filers wih differen consan dephs. As we sweep hrough he objec wih a consan deph plane, he aliasing effec is he wors a he minimum and maximum dephs. The bes rendering qualiy is obained a he opimal deph (Figure 7(b)), no a he focal plane as has been commonly assumed in ligh field [16] or Lumigraph [10] rendering. In fac, he opimal deph can be used as a guidance for selecing he focal plane. For comparison, we also show he rendering resul using aerage deph in Figure 7(c). Similar sweeping effecs hae also been discussed in he dynamically reparameerized ligh field [13]. Howeer, an analyical soluion using he minimum and maximum dephs has neer been presened before. 2.6 Minimum sampling rae for ligh field rendering Wih he aboe heoreical analysis, we are now ready o sole he problem of he minimum sampling rae for ligh field rendering. Since we are dealing wih recangular sampling laice, he Nyquis sampling heorem for 1D signal applies o boh direcions and. According o he Nyquis sampling heorem, in order for a signal o be reconsruced wihou aliasing, he sampling frequency needs o be greaer han he Nyquis rae, or wo imes ha of he Nyquis frequency. Wihou loss of generaliy, we only sudy he Nyquis frequency along he direcion in he frequency domain. Howeer, he Nyquis frequency along he direcion can be analyzed in a similar way. The minimum ineral, by which he replicas of specral suppor can be packed wihou any oerlapping, can be compued as shown in Figure 5(a) where and jp maxp minj = K fh d =2K f fh d (7) h d = 1 z min ; 1 z max K f fh d = min(b s 1=(2) 1=(2)) is he highes frequency for he ligh field signal, which is deermined by he scene exure disribuion (represened by he highes frequency B s ), he resoluion of he sampling camera (), and he resoluion of he rendering camera (). The frequency B s can be compued from he specral suppor of he ligh field. Our formulaion akes he rendering resoluion ino accoun because rendering a a resoluion higher han he oupu resoluion is waseful. For simpliciy, we assume = from now on. The minimum sampling rae is equialen o he maximum camera spacing max, which can be compued as max = 1 K f fh d : (8) The minimum sampling rae can also be inerpreed in erms of he maximum dispariy defined as he projecion error using he opimal reconsrucion filer for rendering. From Equaion 8, we hae he maximum dispariy maxfh d =2= 1 2K f = max( 1=(2B s )): (9) Therefore, he dispariy is less han 1 pixel (i.e., he camera resoluion) or half cycle of he highes frequency (1=B s is defined as a cycle) presened in he EPI image because of he exural complexiy of he obsered scene. If he exural complexiy of he scene is no considered, he minimum sampling rae for ligh field rendering can also be deried in he spaial domain. For example, by considering he ligh field rendering as a synheic aperure opical sysem, we presen an opical analysis of ligh field rendering in Appendix A. The maximum camera spacing will be larger if he scene exure ariaion ges more uniform, or if he rendering camera resoluion becomes lower. By seing he higher frequency par of he specrum o zero so ha B s < 1=(2), we can reduce he minimum sampling rae. One way o reduce B s is o apply a low-pass filer o

7 (a) (b) (c) (d) Figure 7: Sweeping a consan deph plane hrough an objec: (a) a he minimum deph; (b) a he opimal plane; (c) a he aerage disance beween minimum and maximum dephs; (d) a he maximum deph. The bes rendering qualiy is achieed in (b). he inpu - image. This approach is similar o prefilering a ligh field (see Figure 7 in [16]). In paricular, he minimum sampling rae is also deermined by he relaie deph ariaion f(z min ;1 ; z;1 max). The closer he objec ges o he camera, he smaller he z min is, and he higher he minimum sampling rae will be. As f ges larger, he sampling camera will coer a more deailed scene, bu he minimum sampling rae needs o be increased. Therefore, he plenopic sampling problem should no be considered in he image space alone, bu in he join image and geomery space. 3 Minimum sampling in he join image and geomery space In his secion, we will sudy he minimum sampling problem in he join geomery and image space. Since he CPU speed, memory, sorage space, graphics capabiliy and nework bandwidh used ary from users o users, i is ery imporan for users o be able o seek he mos economical balance beween image samples and deph layers for a gien rendering qualiy. I is ineresing o noe ha he minimum sampling rae for ligh field rendering represens essenially one poin in he join image and geomery space, in which lile amoun of deph informaion has been uilized. As more geomerical informaion becomes aailable, fewer images are necessary a any gien rendering resoluion. Figure 8 illusraes he minimum sampling rae in he image space, he minimum sampling cure in he join image and geomery space, and minimum sampling cures a differen rendering resoluions. Any sampling poin aboe he minimum sampling cure (e.g., Figure 8b) is redundan. 3.1 Minimum sampling wih accurae deph From an iniial se of accurae geomerical daa, we can decompose a scene ino muliple layers of sub-regions. Accordingly, he whole specral suppor can be decomposed ino muliple layers (see Figure 5b) due o he correspondence beween a consan deph and is specral suppor. For each decomposed specral suppor, an opimal consan deph filer can be designed. Specifically, for each deph layer i =1 ::: N d, he deph of opimal filer is described as follows = i +(1; i) (10) z i z min z max where i = i ; 0:5 N d Therefore a deph alue can be assigned o one of he deph layers z = z i if ;h d 1 2N d z ; 1 h d : (11) z i 2N d The layers are quanized uniformly in he dispariy space. This is because perspecie images hae been used in he ligh fields. If we use parallel projecion images insead, he quanizaion should be uniform in he deph space [5]. Similar o Equaion 8, he minimum sampling in he join image and accurae deph space is obained when 1 = N d 1 (12) N d K f fh d where N d and are he number of deph layers and he sampling ineral along he direcion, respeciely. The ineral beween replicas is uniformly diided ino N d segmens. The number of deph layers needed for scene represenaion is a funcion of he sampling and rendering camera resoluion, he scene s exure complexiy, he spacing of he sampling cameras and he deph ariaion relaie o he focal lengh Applicaions Based on he aboe quaniaie analysis in he join image and deph space for sufficien rendering, a number of imporan applicaions can be explored. Image-based geomery simplificaion. Gien he appropriae number of image samples an aerage user can afford, he minimum sampling cure in he join space deermines how much deph informaion is needed. Thus, i simplifies he original complex geomerical model o he minimum while sill guaraneeing he same rendering qualiy. Geomery-based image daabase reducion. In conras, gien he number of deph layers aailable, he number of image samples needed can also be reduced o he minimum for a gien rendering resoluion. The reducion of image samples is paricularly useful for ligh field rendering. Leel of deails (LOD) in join image and deph space. The idea of LOD in geomery space can be adoped in our join image and geomery space. When an objec becomes farher away, is relaie size on screen space diminishes so ha he number of required image samples or he number of required deph layers can be reduced accordingly. ooming-in ono and zooming-ou of objecs also demand a dynamic change in he number of image samples or deph layers. Ligh field wih layered deph. A general daa srucure for he minimum sampling cure in he join image and geomery space can be ligh field wih layered deph. Wih differen numbers of images and deph layers used, he rade-off beween rendering speed and daa sorage has o be sudied.

8 Number of images Number of images Number of images Minimum sampling rae Higher Resoluion Number of deph layers Number of deph layers (a) (b) (c) Number of deph layers Figure 8: Plenopic sampling: (a) he minimum sampling rae in image space; (b) he minimum sampling cure in he join image and geomery space (any sampling poin aboe he cure is redundan); (c) minimum sampling cures a differen rendering resoluions. 3.2 Minimum sampling wih deph uncerainy Anoher aspec of minimum sampling in he join image and geomery space is relaed o deph uncerainy. Specifically, minimum sampling wih deph uncerainy describes he quaniaie relaionship beween he number of image samples, noisy deph and deph uncerainy. I is imporan o sudy his relaionship because in general he recoered geomery is noisy as modeling a real enironmen is difficul. Gien an esimaed deph z e and is deph uncerainy, he deph alue should be locaed wihin he range (z e ; ze +). The maximum camera spacing can be compued as max =min z e (z e +)(z e ; ) 2fK f = minze ze 2 ; 2 : (13) 2fK f In addiion, geomerical uncerainy also exiss when an accurae model is simplified. Gien he correc deph z 0 and an esimaed deph z e, he maximum camera spacing can be compued as z ez 0 max = min z e 2fK f jz : (14) e ; z0j Applicaions Knowledge abou he minimum number of images under noisy deph has many pracical applicaions. Minimum sampling rae. For a specific ligh field rendering wih no deph maps or wih noisy deph maps, we can deermine he minimum number of images for anialiased ligh field rendering. Redundan image samples can hen be lef ou from he sampled daabase for ligh field rendering. Rendering-drien ision reconsrucion. This is a ery ineresing applicaion, considering ha general ision algorihms would no recoer accurae scene deph. Gien he number of image samples, how accuraely should he deph be recoered o guaranee he rendering qualiy? Rendering-drien ision reconsrucion is differen from classical geomery-drien ision reconsrucion in ha he former is guided by he deph accuracy ha he rendering process can hae. 4 Experimens Table 1 summarizes he parameers of each ligh field daa se used in our experimens. We assume ha he oupu display has he same resoluion as he inpu image. Furhermore, wihou aking ino consideraion he acual exure disribuion, we assume ha he highes frequency in images is bounded by he resoluion of he capuring camera. We hae used differen seings of focal lengh for he Head, he Saue and he Table. We pu he focal plane slighly in fron of he Head. A smaller focal lengh will reduce he minimum sampling rae. For he Saue, he focal plane is se approximaely a is forehead. In fac, we hae se he focal lengh (3000) ery close o he opimal (3323). Because he Table scene has significan deph ariaions, a small camera focal lengh was used so ha each image can coer a large par of he scene. Firs, we compare he rendering qualiy along he minimal sampling cure in he join image and geomery space, wih he bes rendering qualiy we can obain wih all images and accurae deph. According o our heory (Eq (12)), he number of images is inersely proporional o he number of deph layers in use. The rendering resuls corresponding o fie differen image and deph combinaions along he minimum sampling cure are shown in Figures 11(A)-(E). For example, C(7,8) represens he rendering resul using 7 layers of deph and 8 8 images. In conras, Figure 11(F) shows he bes rendering oupu one can achiee from his se of daa: accurae deph and all images 2. The qualiy of he rendered images along he minimal sampling cure is almos indisinguishable 3 from ha of using all images and accurae deph. Figure 12(a) compares he rendering qualiy using differen layers of deph and a gien number of image samples. Wih 2 2 image samples of he Head, images (A)-(E) in Figure 12(a) show he rendered images wih differen layers of deph a 4, 8, 10, 12, and 24. According o Eq (12), he minimum sampling poin wih 22 images of he Head is a approximaely 12 layers of deph. Noiceable isual arifacs can be obsered when he number of deph is below he minimal sampling poin, as shown in images (A)-(C) of Figure 12(a). On he oher hand, oersampling layers of deph does no improe he rendering qualiy, as shown in he images (D) and (E). Wih he minimal sampling cure, we can now deduce he minimum number of image samples a any gien number of deph layers aailable. For he Table scene, we find ha 3 bis (or 8 layers) of deph informaion is sufficien for ligh field rendering when combined wih image samples (shown in image (D) of Figure 12(b)). When he number of deph layers is below he minimal sampling poin, ligh field rendering produces noiceable arifacs, as shown in images (A)-(C) of Figure 12(b). Gien a single deph layer, our analysis (Eq 12) shows ha he number of images for anialiased rendering of he able scene requires images. Noe ha conenional ligh field may require een a larger number of images wihou using he opimal deph. This ery large se of ligh field daa is due o he signifi- 2 We were no able o use all 6464 images wih accurae deph because of memory limiaions. 3 There exiss lile discrepancy because of he fac ha we can no apply he opimal reconsrucion filer in rendering.

9 (a) (b) Figure 9: Comparison beween conenional ligh field wih images and rendering wih images and 3 bis of deph: (a) arifacs are isible on he lef wih conenional rendering, (b) bu no presen wih addiional geomerical informaion because minimum sampling requiremen is saisfied. can deph ariaions in he Table scene. This perhaps explains why inside-looking-ou ligh field rendering has no been used ofen in pracice. Also according o our analysis, using 3 bis (8 layers) of deph helps o reduce he number of images needed by a facor of 60, o images. For comparison, Figure 9(a) shows conenional ligh field rendering wih images and Figure 9(b) shows he rendering resul wih 1616 images plus 3 bis of deph. Visual arifacs such as double images a he edge of he wall are clearly isible in Figure 9(a). They are no presen in Figure 9(b). Experimens using deph wih uncerainy also demonsrae he effecieness of our analysis. Due o space limiaion, we will no presen any resuls of minimum sampling cure using deph wih uncerainy. 5 Conclusion and fuure work In his paper we hae sudied he problem of plenopic sampling. Specifically, by analyzing he bounds of specral suppor of ligh field signals, we can analyically compue he minimum sampling rae of ligh field rendering. Our analysis is based on he fac ha he specral suppor of a ligh field signal is bounded by only he minimum and maximum dephs, irrespecie of how complicaed he specral suppor migh be because of deph ariaions in he scene. Gien he minimum and maximum dephs, a reconsrucion filer wih an opimal consan deph can be designed for ani-aliased ligh field rendering. The minimum sampling rae for ligh field rendering is obained by compacing he replicas of he specral suppor of he sampled ligh field wihin he smalles ineral. Our work proides a soluion o oercoming he oersampling problem in ligh field capuring and rendering. By sudying plenopic sampling in he join image and geomery space, we hae also deried he minimum sampling cure which quaniaiely describes he relaionship beween he number of images and he informaion on scene geomery, gien a specific rendering resoluion. Indeed, minimum sampling cures wih accurae deph and wih noisy deph sere as he design principles for a number of applicaions. Such ineresing applicaions include image-based geomery simplificaion, geomery-assised image daase reducion, rendering-drien ision reconsrucion, in addiion o deph-assised ligh field compression, or he minimum sampling rae for ligh field rendering. While we hae sudied minimum sampling using ligh fields in his paper, he ery idea of plenopic sampling is also applicable o oher IBR sysems, e.g. concenric mosaics, layered-deph image, iew inerpolaion, and image warping, o name a few. Wih plenopic sampling, here are a number of exciing areas for fuure work. For example, we hae used deph alue in his paper o encode he geomery informaion. Deph is also used in image-assised geomery simplificaion. Howeer, no surface normal has been considered. In he fuure, we would like o experimen wih differen echniques o generae image-assised geomery simplificaion using geomerical represenaions oher han deph. We plan o incorporae he surface normal ino image-based polygon simplificaion. The efficiency of geomery simplificaion can be furher enhanced by considering he sandard echniques in geomerical simplificaion, e.g. isibiliy culling. Anoher ineresing line of fuure work is on how o design a new rendering algorihm for he join image and geomery represenaion. The complexiy of he rendering algorihm should be proporional o he number of deph in use. In addiion, error-bounded deph reconsrucion should be considered as an alernaie o radiional ision reconsrucion, if he reconsrucion resul is o be used for rendering. Gien he error bounds ha are olerable by he rendering algorihms, he difficuly of ision reconsrucion can be much alleiaed. Lasly, we plan o sudy iew-dependen plenopic sampling. Curren analysis of plenopic sampling is based on he assumpion ha he surface is diffuse and lile iew-dependen ariance can occur. I is conceiable ha iew dependen surface propery will increase he minimum sampling rae for ligh field. 6 Acknowledgemens The auhors benefied from discussions wih houchen Lin on runcaed models and Tao Feng on opical analysis of ligh field rendering. Mr. Yin Li s incredible help on preparing Siggraph ideo is grealy appreciaed. The las auhor also wishes o hank Pa Hanrahan for his helpful discussion on he minimum sampling of concenric mosaics while isiing Sanford in April Finally, he auhors hank Siggraph reiewers commens which hae remendously helped o improe he final manuscrip.

10 focal plane synheic aperure lens camera plane camera i+1 film circle of confusion d aperure D max op camera i min focal 0 lengh f Figure 10: A discree synheic aperure opical sysem for ligh field rendering. References [1] E. H. Adelson and J. Bergen. The plenopic funcion and he elemens of early ision. In Compuaional Models of Visual Processing, pages MIT Press, Cambridge, MA, [2] M. Bass, edior. Handbook of Opics. McGraw-Hill, New York, [3] R. C. Bolles, H. H. Baker, and D. H. Marimon. Epipolar-plane image analysis: An approach o deermining srucure from moion. Inernaional Journal of Compuer Vision, 1:7 55, [4] E. Camahor, A. Lerios, and D. Fussell. Uniformly sampled ligh fields. In Proc. 9h Eurographics Workshop on Rendering, pages , [5] J.-X. Chai and H.-Y. Shum. Parallel projecions for sereo reconsrucion. In Proc. CVPR 2000, [6] C. Chang, G. Bishop, and A. Lasra. Ldi ree: A hierarchical represenaion for image-based rendering. SIGGRAPH 99, pages , Augus [7] S. Chen and L. Williams. View inerpolaion for image synhesis. Compuer Graphics (SIGGRAPH 93), pages , Augus [8] T. Feng and H.-Y. Shum. An opical analysis of ligh field rendering. Technical repor, Microsof Research, MSR-TR , May [9] B. Girod. Moion compensaion: isual aspecs, accuracy, and fundamenal limis. In Moion Analysis and Image Sequence Processing. Kluwer, [10] S. J. Gorler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen. The lumigraph. In Compuer Graphics Proceedings, Annual Conference Series, pages 43 54, Proc. SIGGRAPH 96 (New Orleans), Augus ACM SIGGRAPH. [11] M. Halle. Holographic sereograms as discree imaging sysems. In Proc. SPIE Vol.2176, Pracical Holography VIII, pages 73 84, May [12] V. Hlaac, A. Leonardis, and T. Werner. Auomaic selecion of reference iews for image-based scene represenaions. In Proc. ECCV, pages , [13] A. Isaksen, L. McMillan, and S. Gorler. Dynamically reparameerized ligh fields. Technical repor, Technical Repor MIT-LCS-TR-778, May [14] S. Kang. A surey of image-based rendering echniques. In VideoMerics, SPIE Vol. 3641, pages 2 16, [15] J. Lengyel. The conergence of graphics and ision. Technical repor, IEEE Compuer, July [16] M. Leoy and P. Hanrahan. Ligh field rendering. In Compuer Graphics Proceedings, Annual Conference Series, pages 31 42, Proc. SIGGRAPH 96 (New Orleans), Augus ACM SIGGRAPH. [17].-C. Lin and H.-Y. Shum. On he numbers of samples needed in ligh field rendering wih consan-deph assumpion. In Proc. CVPR 2000, [18] W. Mark, L. McMillan, and G. Bishop. Pos-rendering 3d warping. In Proc. Symposium on I3D Graphics, pages 7 16, [19] L. McMillan and G. Bishop. Plenopic modeling: An image-based rendering sysem. Compuer Graphics (SIGGRAPH 95), pages 39 46, Augus [20] H. Schirmacher, W. Heidrich, and H. Seidel. Adapie acquisiion of lumigraphs from synheic scenes. In Eurographics 99, pages , Sep [21] S. M. Seiz and C. M. Dyer. View morphing. In Compuer Graphics Proceedings, Annual Conference Series, pages 21 30, Proc. SIGGRAPH 96 (New Orleans), Augus ACM SIGGRAPH. [22] J. Shade, S. Gorler, L.-W. He, and R. Szeliski. Layered deph images. In Compuer Graphics (SIGGRAPH 98) Proceedings, pages , Orlando, July ACM SIGGRAPH. [23] H.-Y. Shum and L.-W. He. Rendering wih concenric mosaics. In Proc. SIG- GRAPH 99, pages , [24] P. P. Sloan, M. F. Cohen, and S. J. Gorler. Time criical lumigraph rendering. In Symposium on Ineracie 3D Graphics, pages 17 23, Proidence, RI, USA, [25] A. Tekal. Digial Video Processing. Prenice Hall, A An opical analysis of ligh field rendering Similar o [16, 13], we consider he ligh field rendering sysem as a discree synheic aperure opical sysem, as shown in Figure 10. Analogous o he Gaussian opical sysem, we can define he following opical parameers: Focal lengh f; Smalles resolable feaure (on he image plane) d; Aperure D. Disance beween wo adjacen cameras; Circle of confusion c = d=f; Hyperfocal disance D H = D=c. Le he plane of perfec focus be a he disance z op, he minimum and maximum disances a which he rendering is accepable be z min and z max, respeciely. The following relaions exis ([2], ol. 1, p.1.92) z min = which lead o, DHzop and z max = DHzop D H + z op D H ; zop 1 1 = ( + 1 )=2 z op z min z max 1 1 = ( ; 1 )=2 D H z min z max Therefore, o hae he bes rendering qualiy, no maer which opical sysem is used, he focus should be always a z op. Moreoer, o guaranee he rendering qualiy, D H has o be saisfied, i.e., D d=f =( 1 ; 1 )=2 (15) z min z max In oher words, gien he minimum and maximum disances, he maximum camera spacing can be deermined in order o mee he specified rendering qualiy. The hyperfocal disance describes he relaionship among he rendering resoluion (circle of confusion), he scene geomery (deph of field) and he number of images needed (synheic aperure). Inuiiely, he minimum sampling rae is equialen o haing he maximum dispariy less han he smalles resolable feaure on he image plane, e.g, camera resoluion or one pixel, i.e., d = =1. The same resul was also obained by Lin and Shum [17] using a geomerical approach. Equaion 15, no surprisingly, is almos exacly he same as Equaion 8 because D H = 2=h d. Howeer, our approach using specral analysis of ligh field signals incorporaes he exural informaion in he sampling analysis. More deailed opical analysis of ligh field rendering can be found in [8].

11 Focal Maximum Minimum (u ) (s ) Pixels Image Spacing lengh deph deph ineral ineral per image per slab max Head Saue Table Table 1: A summary of parameers used in hree daa ses in our experimens. A(2,32) B(4,16) C(7,8) D(13,4) E(25,2) F(accurae deph,32) Number of Images 35 A(2,32) 30 F(accurae deph,32) B(4,16) 10 C(7,8) 5 D(13,4) E(25,2) Number of Deph Layers Figure 11: Minimum sampling cure for he objec Saue in he join image and geomery space wih accurae geomery. Sampling poins in he figure hae been chosen o be slighly aboe he minimum sampling cure due o quanizaion.

12 Rendered image A(4,2) Rendered image A(8,4) B(8,2) C(10,2) B(8,6) C(8,8) D(12,2) E(24,2) D(8,16) E(8,32) Number of Images 16 Number of Images E(8,32) A(4,2) C(10,2) 2 B(8,2) D(12,2) E(24,2) Number of Deph Layers (a) D(8,16) C(8,8) B(8,6) A(8,4) Number of Deph Layers (b) Figure 12: Minimum sampling poins in he join image and geomery space: (a) for he objec Head, when he number of images is 2 2; (b) for he Table scene, when he number of deph layers is 8.