1 : 2D?3D 15 ; 3D 2D 3D 3D 2D?3D [3] 2D 3D 1 2D?3D (1) ; 1 2D?3D 11 2 : (binocularstereoscopicrenderingbsr) 3D ; [4-5] 3D (depth image-basedrenderingd

Transcription

1 Vol24No1 JournalofComputer-AidedDesign & ComputerGraphics Jan2012 2D?3D ( ) (wliu@nlpriaaccn) :3D 3D 2D 3D 2D?3D 4 : 2D 3D; ; :TP391 ASurveyof2Dto3DConversionTechnologyforFilm Liu WeiWuYihongandHuZhanyi (NationalLaboratoryof Patern RecognitionInstituteof Automation Chinese Academyof SciencesBeijing ) Abstract:3D films have become more and more popular dueto their exceptionalvisualquality enhancementincomparison withthetraditional2doneshoweverduetothehighcostin3dfilm productionwhichseverelyhampersitsfastdevelopmenttheconversiontechnologyfrom 2Dto3D becomesapromisingwaytoaleviatetheprobleminthispaperthevarious2dto3dfilmconversion technologiesaresummarizedmorespecificalytheclassicalconversionapproachesarefirstreviewed accordingtothefourkindsofrelativemotionbetweencameraandscenethenseveralmoreadvanced onesarediscussedfinalysomepromisingfuturedirectionsarelisted Keywords:2Dto3Dconversiontechnologyforfilm;depthmap;paralax;stereoscopicimaging D 3D 3D DreamWorks Animation Jefrey Katzenberg 3D 70 3D [1-2] 3D 3D ( ) 2D 3D : ; : : ( ) (1983 ) 3D ; (1973 ) ; (1961 )

2 1 : 2D?3D 15 ; 3D 2D 3D 3D 2D?3D [3] 2D 3D 1 2D?3D (1) ; 1 2D?3D 11 2 : (binocularstereoscopicrenderingbsr) 3D ; [4-5] 3D (depth image-basedrenderingdibr) [6-8] 3D DIBR 2 ( 64mm) ( 3D [9] 3 : ) 3D 3D 2D 3DTV3D 3D 2 2 3D 2D DIBR 3D 3D 1 2 BSR 1 2 DIBR BSR [10] x 2 B ; z I 1 I 2 (xy) O 1 O 2 2 P l P r P f D= P l-p r z= Bf (1) P l-p r 12 2D?3D 2D?3D 2D 3D 2D?3D 4 : 1) ; 2) ; 3) ; 4)

3 D?3D D max=d maxfordisplay Cue1 (1-Cue2) (1-Cue3) (2) D?3D ; D maxfordisplay DepthFrom Motion:DIBR [1114] ;BSR [412] ; [13] MPEG4 [15] StructureFrom Motion:BSR [16] ;DIBR [17] [18] [19] [20-21] [22] ; S [23] [24] [25-27] [28] [29-31] [32-34] D?3D 211 [13] MPEG4 MPEG4 (depthfrom motiondfm) DFM : DFM 2D?3D ; 2 [15] [411-12] [13] 2 [35] [36] [14] K [11-12] KLT [12] BSR Cue 1 Cue 2 Cue 3 [13] MPEG4 MPEG4 JPEG 4 4 1?4 : [4] USUAN ;

4 1 : 2D?3D 17 ( ); ; ( ) ( ) 2 1 3D [17] DIBR SFM DFM 3D ; DFM BSR 3D [16] SFM (imagebeasedrenderingibr) DFM D SFM ; (structurefrom motionsfm) 2 3D 2 [37] SFM 2D?3D 1 :

5 18 24 F 1 F 2 P stereo 3 F 2 SFM IBR F 1 ; IBR H 3D m original m stereo m stereo=hm original (3) M m stereo H m stereo=p stereom (4) m stereo m original 4 H 4 H BSR [19] ; 2 π ; F 2 F 2 F 1 π X F 1 m F 2 Hm m=hm (5) (5) SFM DIBR [18] Delaunay ; 4 SFM BSR

6 1 : 2D?3D 19 [20] ; ; [21] ; [38-39] [38] 3D 2D?3D [22] : 3 2D [22] 3 ; : ( ) 6 5 u f v 1 f =1 u +1 (6) v ; 0 255? Width (Width ) [40] Hough ; 6

7 P P [27] R ; R u g(xy)=h(xy) f(xy) (7) h(xy)= 1 e - x 2 +y 2 2πσ 2 2σ 2 ( 8) g(xy) f h(xy) σ [28] σ=krk>0 σ u ; ; ; 7 3D ; DIBR ; DFM SFM ; [23] 2 ( S ) (7) [29] 2D?3D 2 2 S ( ) 2 : [24] ; 2 : ; [25-27] 2 3D [41] [25-26]

8 1 : 2D?3D 21 [29] D?3D [29] DFM ;SFM ; ; [30] 2D?3D DFM SFM 3 :1) SFM 3D ; ;2) DFM 2 ;3) [32] : F n F k (n-δ k n+δ) F n H nk m i n=h nk m i k (9) m i n m i k n k 215 Δ 8 F n [42] F n

9 DFM 2 ; [45] [46] [ ] [33] F(xy) DFM F(xy) max F(xy) min DFM M(xy)= ( b-a) [F(xy)-F(xy) min ] +a F(xy) max-f(xy) min (10) 2D?3D ab G(xy) 215 A(xy) 4 2D?3D 烄 M(xy)A(xy)=255 D(xy)= (11) 烅烆 G(xy)A(xy) =0 2 [34] 2 DFM( ) DFM( ) SFM(DIBR) SFM(BSR)

10 1 : 2D?3D 23 2 [47] 9 [47] 2D?3D 9 [29] 10b 2 ; 2 DFM DFM [11] 10c 10a ; 10d [40] 10c :

11 24 24 ; 10e [33] DFM 2 [7] DFM 2D?3D 22 2D?3D 221 2D?3D 2D?3D (layereddepthvideoldv) 3D ; 2 2D?3D 3 : 1 2 Harman [48] (MLA) [51] 2 : ; 3 [52] Barenbrug [53] LDV (α ) MLA ; [54] ; α [49] 223 2D?3D KLT 222 3D BSR 360 3D 3D [55] 2D?3D 3D [56] 3D 3D DIBR 2D?3D DIBR 2D [50] ; 2 3D DIBR

12 1 : 2D?3D 25 3D 2D?3D [47] [56] 2 2 : [57] 3D 3D 3D [55] 3D 3D (consumerelectronicsce) 3D 3D 3D 2D?3D 3D 3D 3D 2D?3D 2D?3D 3 2D?3D 3D 11 ( C7000 NX810 ZF1 3D ; LV ) 3D ( SCH 3D SH ) 3D [58-59] 3D 3D JVC IF-2D3D1 [60] ; 3D MediaCluster ; 2D?3D DDD(Dynamic 3D DigitalDepth) TriDef-3D Experience 3D 3D (Cyberlink) CES2010 (Consumer 2D?3D ElectronicsShow) 3D PowerDVD10 3D 2D 3D 3D 3D 11 3D 3D 3D 2D?3D 2 2 2D

13 26 24 ( 3D (3):3-6) [31] 3D 2 2 Engineering2007 (inchinese) ( ; [D] 2007) ; ( 2D?3D 2009(2): ) UniversityofNorthCarolinaatChapelHil1997 Broadcasting200551(2): D?3D InstrumentationEngineersPress : D 3D stereoscopic conversion ( ) (References): [J] 2008 [3]MatsumotoYTerasakiHSugimoto KetalConversion systemofmonocularimagesequencetostereousing motion paralax[c]?proceedingsofspiebelingham:societyof Photo-OpticalInstrumentationEngineersPress : [4]ZhangJianhuaA studyofpseudothree-dimensionalvideo reconstructionbasedontwo-dimensionalvideosequence[d] Chongqing:ChongqingUniversityColegeofCommunication : [5]Li Liming Li Dahai Shi Lin A new method of 3D conversionfrom2dvideo[j]journalofchengduuniversity ofinformationtechnology2009(2): (inchinese) [J] [6]McMilan L Pizer R S An image-based approach to three-dimensional computer graphics [D] Chapel Hil: [7]ZhangLTam W JStereoscopicimagegenerationbasedon depth images for 3D TV [J] IEEE Transactions on [8]FehnCDepth-image-basedrendering (DIBR)compression andtransmission for a new approach on 3D-TV [C]? ProceedingsofSPIEBelingham:SocietyofPhoto-Optical [9]HarmanPFlackJFoxSetalRapid2D-to-3Dconversion [C]?ProceedingsofSPIEBelingham:Society ofphoto- OpticalInstrumentationEngineersPress :78-86 [10]BerretyR P MRiemens A KMachadoP EReal-time embeddedsystem forstereovideoprocessingfor multiview displays [C]?ProceedingsofSPIEBelingham:Societyof Photo-OpticalInstrumentationEngineersPress : [11]LiTDaiQ HXieX DAneficientmethodforautomatic [C]?Proceedings of the 5th InternationalConferenceon VisualInformationEngineering LosAlamitos:IEEEComputerSocietyPress2008: [12]Kim DMin DSohn KA stereoscopicvideogeneration methodusingstereoscopicdisplaycharacterizationandmotion analysis[j]ieee TransactionsonBroadcasting (2): [1]Li Hongshan Li Shuping Development and analysis of [13]IdesesIYaroslavskyLPFishbainBReal-time2Dto3D digital stereoscopic film [J] Advanced Motion Picture videoconversion[j]journalofreal-timeimageprocessing Technology2008(6):9-12 (inchinese) 2007 (2):3-9 ( [J] [14]ZhangLLawrenceBWangD MetalComparisonstudy 2008(6):9-12) onfeaturematchingandblock matchingforautomatic2dto [2]Sun Yanlu New progress of 3D movie technology [J] 3D video conversion [C]?Proceedings of the 2nd IEE Advanced Motion Picture Technology2008(3):3-6 (in European Conference on Visual Media Production Los Chinese) Alamitos:IEEEComputerSocietyPress2005:

14 1 : 2D?3D 27 [15]Chang Y LFang C Y Ding L Fetal Depth map generationfor2d-to-3d conversion by short-term motion assisted color segmentation [C]?Proceedings of IEEE International Conference on Multimedia and Expo Los Alamitos:IEEEComputerSocietyPress2007: [16]Knorr S Smolic A Sikora T From 2D-to stereo-to multi-viewvideo[c]?proceedingsof3dtv ConferenceLos Alamitos:IEEEComputerSocietyPress2007:1-4 [17]LiPFarinDSKleinG RetalOncreatingdepth maps from monoscopic video using structure from motion [OL]htp:?citeseerxistpsuedu?viewdoc?download doi= &rep=repl&type=pdf [18]Rotem E Wolowelsky K Pelz D Automatic video to stereoscopic video conversion [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress : [19]ZhangG FHua WQin X YetalStereoscopicvideo synthesisfromamonocularvideo[j]ieee Transactionson VisualizationandComputerGraphics200713(4): [20]TsaiY MChangY LChenLGBlock-basedvanishingline andvanishing point detection for 3D scene reconstruction [C]?ProceedingsofInternationalSymposium oninteligent SignalProcessingandCommunicationsLosAlamitos:IEEE ComputerSocietyPress2006: [21]BatiatoSCurtiSCasciaM LetalDepthmapgeneration by image classification [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress : [22]JungYJBaikAKimJetalAnovel2D-to-3Dconversion technique based on relative height-depth cue [C]? ProceedingsofSPIEBelingham:SocietyofPhoto-Optical InstrumentationEngineersPress :72371U [23]Subbarao MSuryaGDepthfromdefocus:aspatialdomain approach [J]InternationalJournal of Computer Vision (3): [24]GuoGZhangNHuoLSetal2Dto3Dconvertionbased onedgedefocusandsegmentation[c]?proceedingsofieee InternationalConference on AcousticsSpeech and Signal ProcessingLos Alamitos:IEEE ComputerSociety Press 2008: [25]Kim CParkJLeeJetalFastextractionofobjectsof interestfrom images with low depth offield [J]ETRI Journal200729(3): [26]Park J Kim C Extracting focused object from low depth-of-fieldimagesequences [C]?ProceedingsofSPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress : [27]KoJKim CKim M 2D-To-3Dstereoscopicconversion: depth-map estimation in a 2D single-view image [C]? ProceedingsofSPIEBelingham:SocietyofPhoto-Optical InstrumentationEngineersPress :66962A [28]VarekampCVandewalePdePuter MetalQuestion interfacefor3dpicturecreationonanautostereoscopicdigital pictureframe [C]?Proceedingsof3DTV ConferenceLos Alamitos:IEEEComputerSocietyPress2009:1-4 [29]SaxenaAChungS HNg A Y3D depthreconstruction from a single stil image [J]International Journal of ComputerVision200776(1):53-69 [30]LiuBYGouldSKolerDSingleimagedepthestimation from predicted semanticlabels [C]?Proceedings ofieee ConferenceonComputerVisionandPaternRecognitionLos Alamitos:IEEEComputerSocietyPress2010: [31]Gutmann MWolfLCohen-OrDSemi-automaticstereo extractionfrom videofootage [C]?Proceedingsofthe12th IEEE International Conference on Computer Vision Los Alamitos:IEEEComputerSocietyPress2009: [32]KunterMKnorrSKrutz AetalUnsupervisedobject segmentationfor2dto3d conversion [C]?Proceedingsof SPIEBelingham:SocietyofPhoto-OpticalInstrumentation EngineersPress :72371B [33]HuangXJWangL HHuangJJetalAdepthextraction methodbasedonmotionandgeometryfor2dto3dconversion AidedDesign & ComputerGraphics201022(6): (inchinese) ( SFM [J] (6): ) [38]ZhangGFJiaJYWongT TetalConsistentdepthmaps recoveryfrom avideosequence [J]IEEE Transactionson Patern Analysisand MachineInteligence200931(6): [C]?Proceedings ofthe 3rd InternationalSymposium on Inteligent Information Technology Application Los Alamitos:IEEEComputerSocietyPress2009: [34]ChangY LChen W YChangJ YetalPrioritydepth fusionforthe2d-to-3dconversionsystem [C]?Proceedings ofspiebelingham:societyofphoto-opticalinstrumentation EngineersPress : [35]TanZCLuR HApplicationofimprovedgenetick-means clusteringalgorithminimagesegmentation[c]?proceedings ofthe1stinternational WorkshoponEducation Technology andcomputersciencelosalamitos:ieeecomputersociety Press2009: [36]DiStefanoLBulgareliAAsimpleandeficientconnected components labeling algorithm [C]?Proceedings of InternationalConferenceonImage AnalysisandProcessing LosAlamitos:IEEEComputerSocietyPress1999: [37]ZhangJian3Dfacialexpressionreconstructionfromvideovia SFManddynamictexturemapping[J]JournalofComputer [39]KangSBSzeliskiRExtractingview-dependentdepthmaps from a colection ofimages [J]InternationalJournal of ComputerVision200458(2): [40]Yang ZhiqiangZheng Fangxuan Depth estimation from singleimage based on vanishing point [J] Journal of InformationTechnologyandApplications20061(3): (inchinese)

15 28 24 ( [51]BarenbrugBBerrety R P MGunnewiek R KRobust [J] 20061(3): ) [41]GeleGColas MDelaunay GHigherorderstatisticsfor detectionandclassificationoffaultyfanbeltsusingacoustical analysis [C]?Proceedings of IEEE Signal Processing Workshopon Higher-OrderStatisticsLos Alamitos:IEEE ComputerSocietyPress1997:43-46 [42]FarinDde WithP H NEfelsberg W Robustbackground estimationforcomplexvideosequences[c]?proceedingsof IEEE International Conference on Image Processing Los Alamitos:IEEEComputerSocietyPress20031: [43]ChengCCLiCTTsaiY MetalHybriddepthcueingfor 2D-to-3D conversion system [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress : [44]ChengC CLiC TChen L GA 2D-to-3D conversion system using edge information [C]?Proceedings of International Conference on Consumer Electronics Los Alamitos:IEEEComputerSocietyPress2010: [45]ChengC CLiC T Huang P SetalA block-based 2D-to-3D conversion system with bilateral filter [C]? Proceedings of International Conference on Consumer ElectronicsLos Alamitos:IEEE ComputerSociety Press 2009:1-3 [46]Paris S Kornprobst P Tumblin Jet al A gentle introductiontobilateralfilteringanditsapplications [C]? ComputerGraphicsProceedingsAnnualConferenceSeries ACM SIGGRAPHNew York:ACM Press2007:1-45 [47]FehnCKaufPOpdeBeeckOetalAnevolutionaryand optimised approach on 3D-TV [C]?Proceedings of International Broadcast Convention Los Alamitos:IEEE ComputerSocietyPress2002: [48]HarmanP VFlack JFox SetalRapid 2D to 3D conversion[c]?proceedingsofspiebelingham:societyof Photo-OpticalInstrumentationEngineersPress : [49]WuC LEr G HXie X DetalA novel methodfor semi-automatic2dto3dvideoconversion [C]?Proceedings of3dtv ConferenceLosAlamitos:IEEEComputerSociety Press2008:65-68 [50]NgK TZhuZ YChanSCAnapproachto2D-To-3D conversionformultiviewdisplays[c]?proceedingsofthe7th InternationalConferenceonInformationCommunicationsand SignalProcessingLos Alamitos:IEEE Computer Society Press2009:1-5 imagedepthand occlusion generation from uncalibrated stereo [C]?Proceedings ofspiebelingham:society of Photo-OpticalInstrumentationEngineersPress : 68031J [52]Gunnewiek R KBerrety R P MBarenbrug Betal Coherentspatialandtemporalocclusion generation [C]? ProceedingsofSPIEBelingham:SocietyofPhoto-Optical InstrumentationEngineersPress : [53]BarenbrugBDeclipse2:multi-layerimageanddepth with transparency made practical [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress :72371G [54]VandewalePGunnewiek R KVarekamp CImproving depth maps withlimited userinput [C]?Proceedings of SPIEBelingham:SocietyofPhoto-OpticalInstrumentation EngineersPress : [55]Lang MHornung AWang OetalNonlineardisparity mapping for stereoscopic 3D [C]?Computer Graphics ProceedingsAnnualConferenceSeriesACM SIGGRAPH New York:ACM Press2010:1-10 [56]KimJBaikAJungYJetal2D-to-3Dconversionbyusing visual atention analysis [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress : [57]LiuTSunJZhengN NetalLearningtodetectasalient object[c]?proceedingsofieee Conferenceon Computer Vision and Patern Recognition Los Alamitos: IEEE ComputerSocietyPress2007:1-8 [58]Tech G Smolic A Brust Het al Optimization and comparisonofcoding algorithmsfor mobile 3DTV [C]? Proceedings of 3DTV Conference Los Alamitos:IEEE ComputerSocietyPress2009:1-4 [59]Shi Ruizhi Summary of grating stereoscopic printing technology[j]chinaprintingandpackagingstudy20091 (5):1-9 (inchinese) ( [J] 20091(5):1-9) [60]FlackJHarroldJWoodgateGJA prototype3d mobile phone equipped with a next-generation autostereoscopic display [C]?ProceedingsofSPIEBelingham:Society of Photo-OpticalInstrumentationEngineersPress : 64900M