ISSN 1000-9825, CODEN RUXUEW E-mail: jos@iscasaccn Journal of Software, Vol18, No6, June 2007, pp1531 1542 http://wwwjosorgcn DOI: 101360/jos181531 Tel/Fax: +86-10-62562563 2007 by Journal of Software All rights reserved GPU 1,3+, 1 1,2, 1 (, 100080) 2 (, ) 3 (, 100049) Real-Time Rendering Depth Images on GPU by Forward Warping LIU Bao-Quan 1,3+, LIU Xue-Hui 1, WU En-Hua 1,2 1 (State Key Laboratory of Computer Science, Institute of Software, The Chinese Academy of Sciences, Beijing 100080, China) 2 (Department of Computer and Information Science, Faculty of Science and Technology, University of Macau, Macao) 3 (Graduate School, The Chinese Academy of Sciences, Beijing 100049, China) + Corresponding author: Phn: +86-10-62522028, Fax +86-10-62563894, E-mail: lbq@iosaccn, http://lcsiosaccn/~lbq/ Liu BQ, Liu XH, Wu EH Real-Time rendering depth images on GPU by forward warping Journal of Software, 2007,18(6):1531 1542 http://wwwjosorgcn/1000-9825/18/1531htm Abstract: This paper presents a new pipeline for rendering depth images entirely on GPU (graphics processing unit) The implementation exploits inherent parallelism of GPU to speed up the rendering of depth images By the scheme, a novel forward 3D warping method is proposed for vertex shader to obtain high rendering performance Furthermore, the hardware pipeline s rasterization function is utilized to conduct the image re-sampling efficiently to generate holes free rendering results Per pixel lighting effect is implemented in pixel shader to get high image quality The rendering shows rapid performance at full screen resolution, with correct self-occlusions and accurate silhouettes Moreover, a real-time walkthrough system is implemented for the objects based on cylindrical depth image rendered by view-dependent dynamic LOD (level of detail) representation at runtime Key words: : graphics hardware; GPU (graphics processing unit); real-time rendering; depth image; image-based rendering (IBR); per-pixel lighting GPU(graphics processing unit) GPU vertex shader,,,, pixel shader,,,, LOD(level of detail) : ;GPU(graphics processing unit); ; ; Supported by the National Natural Science Foundation of China under Grant Nos60473105, 60573155 ( ); the National Basic Research Program of China under Grant No2002CB312102 ( (973)) Received 2005-11-23; Accepted 2006-04-27
1532 Journal of Software Vol18, No6, June 2007 : TP391 : A,,,,,,,, GPU, Macedonia [1] : GPU GPU, GPU,,, vertex shader,,, Relief Texture Mapping(RTM) [2 4] RTM,, 1 RTM, 941%,, GPU ;,, RTM RTM, :, 2 1 X s, X i, 2 (a) Depth image (b) Normal map (c) Color texture (d) Rendering result (a) (b) (c) (d) Fig1 Input of our system is in (a), (b), and (c), the rendering result is shown in (d) 1 (a),(b) (c), (d) X Target image plane X i X s X t Reference image plane Fig2 C s The two-step rendering procedure: Pre-warping followed by conventional texture-mapping 2 :, C t
: GPU 1533,,, : 1) vertex shader, GPU 2) vertex shader 2D,, Z buffer, 3), RTM, RTM 4),,, LOD(level of detail) 1 2 3 4 5 1 Bump mapping [5],, Displacement mapping [6 8] Displacement mapping,, Displacement mapping,, 4 4,,, vertex shader Relief texture mapping(rtm) [2 4] parallax mapping [9], RTM, RTM, (, ) Fujita [10] RTM, reflection mapping, RTM, RTM, RTM, RTM Relief mapping [11],,,,, [12,13],,Donnelly [14],,,, [15,16],, 3, pixel shader, Dmesh [17] (i,j) P ij R; (, P ij );,,, 4 4,,
1534 Journal of Software Vol18, No6, June 2007,Dmesh,,,,, 2 GPU 21 GPU, :(1) ;(2) Z-buffer 1, X s X i,, vertex shader,, pixel shader, 2, 1,, 3 Reference image Pre-Warping in vertex shader Hardware Lighting in + rasterization + pixel shader Intermediate image in Pbuffer Intermediate image in Pbuffer Texture mapping Final image Fig3 The flowcharts of the two rendering passes 3 22 vertex shader GPU,, X s, McMillan [18,19],, vetex shader vetex shader, (u s,v s ) ( u, v),, (u i,v i )=(u s + u,v s + v) 3D warping Image-Based Rendering using Image Warping, [2,18] ( u, v) 4 u s u=u i u s 4, C t FX u i u s X, u/d e =depth/(depth+dt),d e u e u s,d e =u e u s ;D t C t,, D t ;u e (epipolar),u s u e ;depth u s, u:
: GPU 1535 u=(u e u s ) depth/(depth+d t )=(u e u s )/(1+D t /depth)=(u e u s ) Table(depth),,Table(depth)=1/(1+D t /depth), vertex texture vertex shader depth v Dmesh,, (Z-Buffer ),, 1, ; 2,, 4, Z-Buffer, Z t =D t +depth 1, vertex shader Z t, pixel shader, Pbuffer, 2 Z t, ( 17 ), shadow map, Fig4 u s F X u D e u i Top view of depth image pre-warping, (u s,v s ) (u i,v i ),,, 23 :,, ( ) vertex shader,, pixel shader, 24 pixel shader, pixel shader Phong 1, 1(d), : 1) (VD), C t, (u e,v e ), :D t C t (u e,v e, D t ) (u s,v s,depth), VD=(u e,v e, D t ) (u s,v s,depth) 2), : 3) Phong,,, pixel shader, pixel shader Policarpo [11] Donnelly [14], pixel shader,,, C t u e D t
1536 Journal of Software Vol18, No6, June 2007 25 2 : 1 ( ) Pbuffer,, 2,,, 1(d) 17 3 31,,, 2,, [4,20],,, 6 6, ( ), ( RTM 6 ),,,RTM pre-warping, 9 pre-warping, RTM,,,33,, DPCC(double projective cylindrical camera),, ( ), 5(a) ;,, 5(b) (a) Top view (perspective projection) (b) Side view (parallel projection) (a) ( ) (b) ( ) Fig5 Double projective cylindrical camera 5 DPCC, [19] ( 6(a) ),, DPCI(double projective cylindrical image), 6(b) 67 170 DPCIDPCI 6,
: GPU 1537 warping DPCI u s, 7,C ;u s, B; u s u s B depth (a) Image projected by cylindrical pinhole-camera model (a) C B depth u s (b) DPCI projected by DPCC (b) (DPCI) Fig6 Comparison of different cylindrical images Fig7 The pixel s depth value of DPCI 6 32 DPCI 7 DPCI DPCI, : 1 ; 2, 8 DPCI Pre-Warping Intermediate image in tangent plane Texture mapping Final image Fig8 The flowchart of the two-pass rendering 8 9,,, E,E (epipolar) DPCI Tangent plane ; 2,, E, Pre-Warping 10 Texture mapping, DPCI X s X i, 3D warping, B X i X t C t 4, Fig9 X i C t B, X s B X s, X i C s C t Target image plane 2-D illustration of the rendering procedure 9
1538 Journal of Software Vol18, No6, June 2007 Case 2 X s B C s E Tangent plane Case 1 X i Target image plane Case 3 X t C t Fig10 DPCI to tangent plane pre-warping 10 DPCI Fig11 Three cases of the viewpoint s position 11 3 33,, 12(a),,,,, Z-Buffer ( ), 2, DPCI, Z-Buffer ( ), 11, 3 : Case 1, DPCI, ; Case 2, DPCI, 12 ; Case 3, DPCI,,, : DPCI; (a) Rendered by a DPCI (b) Rendered by a top planar depth image (c) The composition of the first two (a) DPCI (b) (c) Fig12 34 DPCI Rendering DPCI with a top end cap 12 DPCI (meso-structure),, III (tree man) : DPCI, DPCI, pixel shader
: GPU 1539, DPCI,, 13 35 LOD Fig13 Meso-Structure added to DPCI 13 DPCI DPCI N N, LOD,,,, DPCI, Dist object2view Iter step DPCI, DPCI, Iter step, Dist object2view A, A Iter step,iter step 1,,N N;, Nd Nd, Nd=N Iter step,nd<n, DPCI LOD, DPCI, 14, DPCI Fig14 Real-Time walk through system in a scene full of image-based objects at 48327 fps 14 DPCI (48327 fps) 4 OpenGL Cg Windows XP Pentium IV 24 GHz 512M, nvidia GeForceFx5200, 1, 15,
1540 Journal of Software Vol18, No6, June 2007 Table 1 Comparison of frame rate between DPCI based rendering and traditional polygon based rendering 1 DPCI based Polygon based Object Polygon number Resolution of DPCI Ratio of fps rendering (fps) rendering (fps) 512 512 67514 2087 Venus 67 170 32351 256 256 185795 5743 512 512 67514 2382 Man s head 78 157 28346 256 256 185795 6554 (a) Traditional polygon method (b) DPCI based method (input DPCI is 256 256) (c) DPCI based method (input DPCI is 512 512) (a) (b) DPCI ( 256 256) (c) DPCI ( 512 512) Fig15 Comparison of the rendering quality 15 : ;, [21],, : (,Per-Pixel displacement mapping [14] relief mapping [11], DMesh [17] ), 2, :1024 768, 16, 512 512,, 16 Table 2 Comparison of frame rate for planar depth image rendering 2 Resolution of Per-Pixel displacement input image mapping [14] Relief mapping [11] (fps) DMesh [17] (fps) Our method (fps) (fps) 512 512 1836 585 17472 6814 17 DPCI, ; DPCI, DPCI, [2,3],DPCI ;,,,, http://lcsiosaccn/~lbq/publicationshtm 5 GPU, RTM,,
: GPU 1541 ( 18 ) Fig16 Rendering results of planar depth image 16 Fig17 The hybrid rendering results of DPCI interpenetrated with standard triangle based geometry teapots 17 DPCI Fig18 A full screen walking-through system based on image-based objects at 41716 fps 18 (41716 fps) References: [1] Macedonia M The GPU enters computing s mainstream IEEE Computer Society, 2003,36(10):106 108 [2] Oliveira MM Relief texture mapping [PhD Thesis] North Carolina: University of North Carolina, 2000 [3] Oliveira MM, Bishop G, McAllister D Relief texture mapping In: Whitted T, ed Proc of the SIGGRAPH 2000 New Orleans: ACM Press, 2000 359 368 [4] Oliveira MM, Bishop G Image-Based objects In: Proc of the 99 ACM Symp on Interactive 3D Graphics Atlanta: ACM Press, 1999 191 198 [5] James B Simulation of wrinkled surfaces Computer Graphics (Proc of the SIGGRAPH 78), 1978,12(3):286 292
1542 Journal of Software Vol18, No6, June 2007 [6] Cook RL Shade trees Computer Graphics (Proc of the SIGGRAPH 84), 1984,18(3):223 231 [7] Hirche J, Ehlert A, Guthe S, Doggett M Hardware accelerated per-pixel displacement mapping In: Proc of the Graphics Interface Waterloo: ACM Press, 2004 153 158 [8] Kautz J, Seidel HP Hardware accelerated displacement mapping for image based rendering In: Proc of the Graphics Interface 2001 Toronto: Canadian Information Processing Society, 2001 61 70 [9] Tomomichi K, Takahei T, Inami M, Kawakami N, Yanagida Y, Maeda T, Tachi S Detailed shape representation with parallax mapping In: Proc of the 11th Int l Conf on Artificial Reality and Telexistence (ICAT 2001) Tokyo: ACM Press, 2001 205 208 [10] Fujita M, Kanai T Hardware-Assisted relief texture mapping In: Navazo I, ed Proc of the Eurographics 2002 Saarbrücken: Eurographics Association, 2002 257 262 [11] Policarpo F, Oliveira MM, Comba JLD Real-Time relief mapping on arbitrary polygonal surfaces ACM Trans on Graphics (Proc of the SIGGRAPH I3D 2005), 2005,24(3):155 162 [12] Yerex K, Jagersand M Displacement mapping with ray-casting in hardware In: Proc of the SIGGRAPH 2004 Los Angeles: ACM Press, 2004 [13] Delgass LL, McGhee K Parallax searching and mesosurface shadowing In: Proc of the SIGGRAPH 2005 Los Angeles: ACM Press, 2005 [14] Donnelly W GPU Gems 2nd ed, Addison-Wesley, 2005 123 136 [15] Wang LF, Wang X, Tong X, Lin S, Hu SM, Guo BN, Shum HY View-Dependent displacement mapping ACM Trans on Graphics (Proc of the SIGGRAPH 2003), 2003,22(3):334 339 [16] Wang X, Tong X, Lin S, Hu SM, Guo BN, Shum HY Generalized displacement maps In: Proc of the Eurograph Symp on Rendering 2004 Norrköping: Eurographics Association, 2004 227 234 [17] Pajarola R, Sainz M, Meng Y DMesh: Fast depth-image meshing and warping Int l Journal of Image and Graphics (IJIG), 2004, 4(4):1 29 [18] McMillan Jr L An image-based approach to three-dimensional computer graphics [PhD Thesis] University of North Carolina, 1997 [19] McMillan L, Bishop G Plenoptic modeling: An image-based rendering system Computer Graphics (Proc of the SIGGRAPH 95), 1995, 39 46 [20] Adelson EH, Bergen J The plenoptic function and the elements of early vision In: Computational Models of Visual Processing Cambridge: MIT Press, 1991 3 20 [21] Li KY, Wang WC, Wu EH Bar texture mapping Journal of Software, 2004,15:179 189 (in Chinese with English abstract) : [21],,,2004,15:179 189 (1975 ),,,,,, (1947 ),,,,CCF,,, (1968 ),,,,CCF,,