24 1 2012 1 Vol24No1 JournalofComputer-AidedDesign & ComputerGraphics Jan2012 2D?3D ( 100190) (wliu@nlpriaaccn) :3D 3D 2D 3D 2D?3D 4 : 2D 3D; ; :TP391 ASurveyof2Dto3DConversionTechnologyforFilm Liu WeiWuYihongandHuZhanyi (NationalLaboratoryof Patern RecognitionInstituteof Automation Chinese Academyof SciencesBeijing 100190) Abstract:3D films have become more and more popular dueto their exceptionalvisualquality enhancementincomparison withthetraditional2doneshoweverduetothehighcostin3dfilm productionwhichseverelyhampersitsfastdevelopmenttheconversiontechnologyfrom 2Dto3D becomesapromisingwaytoaleviatetheprobleminthispaperthevarious2dto3dfilmconversion technologiesaresummarizedmorespecificalytheclassicalconversionapproachesarefirstreviewed accordingtothefourkindsofrelativemotionbetweencameraandscenethenseveralmoreadvanced onesarediscussedfinalysomepromisingfuturedirectionsarelisted Keywords:2Dto3Dconversiontechnologyforfilm;depthmap;paralax;stereoscopicimaging 2010 3D 3D 3D DreamWorks Animation Jefrey Katzenberg 3D 70 3D [1-2] 3D 3D ( ) 2D 3D :2010-11-23; :2011-09-29 : (6107010760835003) (1983 ) 3D ; (1973 ) ; (1961 )
1 : 2D?3D 15 ; 3D 2D 3D 3D 2D?3D [3] 2D 3D 1 2D?3D (1) ; 1 2D?3D 11 2 : (binocularstereoscopicrenderingbsr) 3D ; [4-5] 3D (depth image-basedrenderingdibr) [6-8] 3D DIBR 2 ( 64mm) ( 3D [9] 3 : ) 3D 3D 2D 3DTV3D 3D 2 2 3D 2D DIBR 3D 3D 1 2 BSR 1 2 DIBR BSR [10] x 2 B ; z I 1 I 2 (xy) O 1 O 2 2 P l P r P f D= P l-p r z= Bf (1) P l-p r 12 2D?3D 2D?3D 2D 3D 2D?3D 4 : 1) ; 2) ; 3) ; 4)
16 24 2 2D?3D D max=d maxfordisplay Cue1 (1-Cue2) (1-Cue3) (2) 1 4 3 1 2D?3D ; D maxfordisplay 1 1 2 3 DepthFrom Motion:DIBR [1114] ;BSR [412] ; [13] MPEG4 [15] StructureFrom Motion:BSR [16] ;DIBR [17] [18] [19] [20-21] [22] ; S [23] [24] [25-27] [28] [29-31] [32-34] 4 21 2D?3D 211 [13] MPEG4 MPEG4 (depthfrom motiondfm) DFM : DFM 2D?3D ; 2 [15] [411-12] [13] 2 [35] [36] [14] K [11-12] KLT [12] BSR Cue 1 Cue 2 Cue 3 [13] MPEG4 MPEG4 JPEG 4 4 1?4 : [4] USUAN ;
1 : 2D?3D 17 ( ); 3 2 2 ; ( ) ( ) 2 1 3D [17] DIBR SFM DFM 3D ; DFM BSR 3D [16] SFM (imagebeasedrenderingibr) DFM 3 212 3D SFM ; (structurefrom motionsfm) 2 3D 2 [37] SFM 2D?3D 1 :
18 24 F 1 F 2 P stereo 3 F 2 SFM IBR F 1 ; IBR H 3D m original m stereo m stereo=hm original (3) M m stereo H m stereo=p stereom (4) m stereo m original 4 H 4 H BSR [19] ; 2 π ; F 2 F 2 F 1 π X F 1 m F 2 Hm m=hm (5) (5) SFM DIBR [18] Delaunay ; 4 SFM BSR
1 : 2D?3D 19 [20] ; ; [21] ; [38-39] [38] 3D 2D?3D 5 213 [22] : 3 2D [22] 3 ; : ( ) 6 5 u f v 1 f =1 u +1 (6) v ; 0 255? Width (Width ) [40] Hough ; 6
20 24 7 P P [27] R ; R u g(xy)=h(xy) f(xy) (7) h(xy)= 1 e - x 2 +y 2 2πσ 2 2σ 2 ( 8) g(xy) f h(xy) σ [28] σ=krk>0 σ u ; ; ; 7 3D ; DIBR ; DFM SFM ; [23] 2 ( S ) (7) [29] 2D?3D 2 2 S ( ) 2 : [24] ; 2 : ; [25-27] 2 3D [41] [25-26]
1 : 2D?3D 21 [29] 2 214 3 2D?3D [29] DFM ;SFM ; ; [30] 2D?3D DFM SFM 3 :1) SFM 3D ; ;2) DFM 2 ;3) [32] : 4 2 2 F n F k (n-δ k n+δ) F n H nk m i n=h nk m i k (9) m i n m i k n k 215 Δ 8 F n [42] F n
2 24 8 3 DFM 2 ; [45] [46] [33-3443-45] [33] F(xy) DFM F(xy) max F(xy) min DFM M(xy)= ( b-a) [F(xy)-F(xy) min ] +a F(xy) max-f(xy) min (10) 2D?3D ab G(xy) 215 A(xy) 4 2D?3D 烄 M(xy)A(xy)=255 D(xy)= (11) 烅烆 G(xy)A(xy) =0 2 [34] 2 DFM( ) DFM( ) SFM(DIBR) SFM(BSR)
1 : 2D?3D 23 2 [47] 9 [47] 2D?3D 9 [29] 10b 2 ; 2 DFM DFM 3 2 214 [11] 10c 10a ; 10d [40] 10c : 213 10
24 24 ; 10e [33] DFM 2 [7] DFM 2D?3D 22 2D?3D 221 2D?3D 2D?3D (layereddepthvideoldv) 3D ; 2 2D?3D 3 : 1 2 Harman [48] (MLA) [51] 2 : ; 3 [52] Barenbrug [53] LDV (α ) MLA ; [54] ; α [49] 223 2D?3D KLT 222 3D BSR 360 3D 3D [55] 2D?3D 3D [56] 3D 3D DIBR 2D?3D DIBR 2D [50] ; 2 3D DIBR
1 : 2D?3D 25 3D 2D?3D [47] [56] 2 2 : [57] 3D 3D 3D [55] 3D 3D (consumerelectronicsce) 3D 3D 3D 2D?3D 3D 3D 3D 2D?3D 2D?3D 3 2D?3D 3D 11 ( C7000 NX810 ZF1 3D ; LV ) 3D ( SCH 3D SH ) 3D [58-59] 3D 3D JVC IF-2D3D1 [60] ; 3D MediaCluster ; 2D?3D DDD(Dynamic 3D DigitalDepth) TriDef-3D Experience 3D 3D (Cyberlink) CES2010 (Consumer 2D?3D ElectronicsShow) 3D PowerDVD10 3D 2D 3D 3D 3D 11 3D 3D 3D 2D?3D 2 2 2D
26 24 ( 3D (3):3-6) 108-112 [31] 3D 2 2 Engineering2007 (inchinese) ( ; [D] 2007) ; ( 2D?3D 2009(2):138-141) UniversityofNorthCarolinaatChapelHil1997 Broadcasting200551(2):191-199 2D?3D InstrumentationEngineersPress20045291:93-104 2D 3D 649014 stereoscopic conversion ( ) (References): [J] 2008 [3]MatsumotoYTerasakiHSugimoto KetalConversion systemofmonocularimagesequencetostereousing motion paralax[c]?proceedingsofspiebelingham:societyof Photo-OpticalInstrumentationEngineersPress19973012: [4]ZhangJianhuaA studyofpseudothree-dimensionalvideo reconstructionbasedontwo-dimensionalvideosequence[d] Chongqing:ChongqingUniversityColegeofCommunication : [5]Li Liming Li Dahai Shi Lin A new method of 3D conversionfrom2dvideo[j]journalofchengduuniversity ofinformationtechnology2009(2):138-141 (inchinese) [J] [6]McMilan L Pizer R S An image-based approach to three-dimensional computer graphics [D] Chapel Hil: [7]ZhangLTam W JStereoscopicimagegenerationbasedon depth images for 3D TV [J] IEEE Transactions on [8]FehnCDepth-image-basedrendering (DIBR)compression andtransmission for a new approach on 3D-TV [C]? ProceedingsofSPIEBelingham:SocietyofPhoto-Optical [9]HarmanPFlackJFoxSetalRapid2D-to-3Dconversion [C]?ProceedingsofSPIEBelingham:Society ofphoto- OpticalInstrumentationEngineersPress20024660:78-86 [10]BerretyR P MRiemens A KMachadoP EReal-time embeddedsystem forstereovideoprocessingfor multiview displays [C]?ProceedingsofSPIEBelingham:Societyof Photo-OpticalInstrumentationEngineersPress20076490: [11]LiTDaiQ HXieX DAneficientmethodforautomatic [C]?Proceedings of the 5th InternationalConferenceon VisualInformationEngineering LosAlamitos:IEEEComputerSocietyPress2008:256-260 [12]Kim DMin DSohn KA stereoscopicvideogeneration methodusingstereoscopicdisplaycharacterizationandmotion analysis[j]ieee TransactionsonBroadcasting200854 (2):188-197 [1]Li Hongshan Li Shuping Development and analysis of [13]IdesesIYaroslavskyLPFishbainBReal-time2Dto3D digital stereoscopic film [J] Advanced Motion Picture videoconversion[j]journalofreal-timeimageprocessing Technology2008(6):9-12 (inchinese) 2007 (2):3-9 ( [J] [14]ZhangLLawrenceBWangD MetalComparisonstudy 2008(6):9-12) onfeaturematchingandblock matchingforautomatic2dto [2]Sun Yanlu New progress of 3D movie technology [J] 3D video conversion [C]?Proceedings of the 2nd IEE Advanced Motion Picture Technology2008(3):3-6 (in European Conference on Visual Media Production Los Chinese) Alamitos:IEEEComputerSocietyPress2005:122-129
1 : 2D?3D 27 [15]Chang Y LFang C Y Ding L Fetal Depth map generationfor2d-to-3d conversion by short-term motion assisted color segmentation [C]?Proceedings of IEEE International Conference on Multimedia and Expo Los Alamitos:IEEEComputerSocietyPress2007:1958-1961 [16]Knorr S Smolic A Sikora T From 2D-to stereo-to multi-viewvideo[c]?proceedingsof3dtv ConferenceLos Alamitos:IEEEComputerSocietyPress2007:1-4 [17]LiPFarinDSKleinG RetalOncreatingdepth maps from monoscopic video using structure from motion [OL]htp:?citeseerxistpsuedu?viewdoc?download doi= 10111297308&rep=repl&type=pdf [18]Rotem E Wolowelsky K Pelz D Automatic video to stereoscopic video conversion [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress20055664:198-206 [19]ZhangG FHua WQin X YetalStereoscopicvideo synthesisfromamonocularvideo[j]ieee Transactionson VisualizationandComputerGraphics200713(4):686-696 [20]TsaiY MChangY LChenLGBlock-basedvanishingline andvanishing point detection for 3D scene reconstruction [C]?ProceedingsofInternationalSymposium oninteligent SignalProcessingandCommunicationsLosAlamitos:IEEE ComputerSocietyPress2006:586-589 [21]BatiatoSCurtiSCasciaM LetalDepthmapgeneration by image classification [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress20045302:95-104 [22]JungYJBaikAKimJetalAnovel2D-to-3Dconversion technique based on relative height-depth cue [C]? ProceedingsofSPIEBelingham:SocietyofPhoto-Optical InstrumentationEngineersPress20097237:72371U [23]Subbarao MSuryaGDepthfromdefocus:aspatialdomain approach [J]InternationalJournal of Computer Vision 199413(3):271-294 [24]GuoGZhangNHuoLSetal2Dto3Dconvertionbased onedgedefocusandsegmentation[c]?proceedingsofieee InternationalConference on AcousticsSpeech and Signal ProcessingLos Alamitos:IEEE ComputerSociety Press 2008:2181-2184 [25]Kim CParkJLeeJetalFastextractionofobjectsof interestfrom images with low depth offield [J]ETRI Journal200729(3):353-362 [26]Park J Kim C Extracting focused object from low depth-of-fieldimagesequences [C]?ProceedingsofSPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress20066007:607710 [27]KoJKim CKim M 2D-To-3Dstereoscopicconversion: depth-map estimation in a 2D single-view image [C]? ProceedingsofSPIEBelingham:SocietyofPhoto-Optical InstrumentationEngineersPress20076696:66962A [28]VarekampCVandewalePdePuter MetalQuestion interfacefor3dpicturecreationonanautostereoscopicdigital pictureframe [C]?Proceedingsof3DTV ConferenceLos Alamitos:IEEEComputerSocietyPress2009:1-4 [29]SaxenaAChungS HNg A Y3D depthreconstruction from a single stil image [J]International Journal of ComputerVision200776(1):53-69 [30]LiuBYGouldSKolerDSingleimagedepthestimation from predicted semanticlabels [C]?Proceedings ofieee ConferenceonComputerVisionandPaternRecognitionLos Alamitos:IEEEComputerSocietyPress2010:1253-1260 [31]Gutmann MWolfLCohen-OrDSemi-automaticstereo extractionfrom videofootage [C]?Proceedingsofthe12th IEEE International Conference on Computer Vision Los Alamitos:IEEEComputerSocietyPress2009:136-142 [32]KunterMKnorrSKrutz AetalUnsupervisedobject segmentationfor2dto3d conversion [C]?Proceedingsof SPIEBelingham:SocietyofPhoto-OpticalInstrumentation EngineersPress20097237:72371B [33]HuangXJWangL HHuangJJetalAdepthextraction methodbasedonmotionandgeometryfor2dto3dconversion AidedDesign & ComputerGraphics201022(6):949-958 (inchinese) ( SFM [J] 201022(6):949-958) [38]ZhangGFJiaJYWongT TetalConsistentdepthmaps recoveryfrom avideosequence [J]IEEE Transactionson Patern Analysisand MachineInteligence200931(6): [C]?Proceedings ofthe 3rd InternationalSymposium on Inteligent Information Technology Application Los Alamitos:IEEEComputerSocietyPress2009:294-298 [34]ChangY LChen W YChangJ YetalPrioritydepth fusionforthe2d-to-3dconversionsystem [C]?Proceedings ofspiebelingham:societyofphoto-opticalinstrumentation EngineersPress20086805:680513 [35]TanZCLuR HApplicationofimprovedgenetick-means clusteringalgorithminimagesegmentation[c]?proceedings ofthe1stinternational WorkshoponEducation Technology andcomputersciencelosalamitos:ieeecomputersociety Press2009:625-628 [36]DiStefanoLBulgareliAAsimpleandeficientconnected components labeling algorithm [C]?Proceedings of InternationalConferenceonImage AnalysisandProcessing LosAlamitos:IEEEComputerSocietyPress1999:322-327 [37]ZhangJian3Dfacialexpressionreconstructionfromvideovia SFManddynamictexturemapping[J]JournalofComputer- 974-988 [39]KangSBSzeliskiRExtractingview-dependentdepthmaps from a colection ofimages [J]InternationalJournal of ComputerVision200458(2):139-163 [40]Yang ZhiqiangZheng Fangxuan Depth estimation from singleimage based on vanishing point [J] Journal of InformationTechnologyandApplications20061(3):229-235 (inchinese)
28 24 ( [51]BarenbrugBBerrety R P MGunnewiek R KRobust [J] 20061(3):229-235) [41]GeleGColas MDelaunay GHigherorderstatisticsfor detectionandclassificationoffaultyfanbeltsusingacoustical analysis [C]?Proceedings of IEEE Signal Processing Workshopon Higher-OrderStatisticsLos Alamitos:IEEE ComputerSocietyPress1997:43-46 [42]FarinDde WithP H NEfelsberg W Robustbackground estimationforcomplexvideosequences[c]?proceedingsof IEEE International Conference on Image Processing Los Alamitos:IEEEComputerSocietyPress20031:145-148 [43]ChengCCLiCTTsaiY MetalHybriddepthcueingfor 2D-to-3D conversion system [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress20097237:723721 [44]ChengC CLiC TChen L GA 2D-to-3D conversion system using edge information [C]?Proceedings of International Conference on Consumer Electronics Los Alamitos:IEEEComputerSocietyPress2010:377-378 [45]ChengC CLiC T Huang P SetalA block-based 2D-to-3D conversion system with bilateral filter [C]? Proceedings of International Conference on Consumer ElectronicsLos Alamitos:IEEE ComputerSociety Press 2009:1-3 [46]Paris S Kornprobst P Tumblin Jet al A gentle introductiontobilateralfilteringanditsapplications [C]? ComputerGraphicsProceedingsAnnualConferenceSeries ACM SIGGRAPHNew York:ACM Press2007:1-45 [47]FehnCKaufPOpdeBeeckOetalAnevolutionaryand optimised approach on 3D-TV [C]?Proceedings of International Broadcast Convention Los Alamitos:IEEE ComputerSocietyPress2002:357-365 [48]HarmanP VFlack JFox SetalRapid 2D to 3D conversion[c]?proceedingsofspiebelingham:societyof Photo-OpticalInstrumentationEngineersPress20024660: 78-86 [49]WuC LEr G HXie X DetalA novel methodfor semi-automatic2dto3dvideoconversion [C]?Proceedings of3dtv ConferenceLosAlamitos:IEEEComputerSociety Press2008:65-68 [50]NgK TZhuZ YChanSCAnapproachto2D-To-3D conversionformultiviewdisplays[c]?proceedingsofthe7th InternationalConferenceonInformationCommunicationsand SignalProcessingLos Alamitos:IEEE Computer Society Press2009:1-5 imagedepthand occlusion generation from uncalibrated stereo [C]?Proceedings ofspiebelingham:society of Photo-OpticalInstrumentationEngineersPress20086803: 68031J [52]Gunnewiek R KBerrety R P MBarenbrug Betal Coherentspatialandtemporalocclusion generation [C]? ProceedingsofSPIEBelingham:SocietyofPhoto-Optical InstrumentationEngineersPress20097237:723713 [53]BarenbrugBDeclipse2:multi-layerimageanddepth with transparency made practical [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress20097237:72371G [54]VandewalePGunnewiek R KVarekamp CImproving depth maps withlimited userinput [C]?Proceedings of SPIEBelingham:SocietyofPhoto-OpticalInstrumentation EngineersPress20107524:752414 [55]Lang MHornung AWang OetalNonlineardisparity mapping for stereoscopic 3D [C]?Computer Graphics ProceedingsAnnualConferenceSeriesACM SIGGRAPH New York:ACM Press2010:1-10 [56]KimJBaikAJungYJetal2D-to-3Dconversionbyusing visual atention analysis [C]?Proceedings of SPIE Belingham: Society of Photo-Optical Instrumentation EngineersPress20107524:752412 [57]LiuTSunJZhengN NetalLearningtodetectasalient object[c]?proceedingsofieee Conferenceon Computer Vision and Patern Recognition Los Alamitos: IEEE ComputerSocietyPress2007:1-8 [58]Tech G Smolic A Brust Het al Optimization and comparisonofcoding algorithmsfor mobile 3DTV [C]? Proceedings of 3DTV Conference Los Alamitos:IEEE ComputerSocietyPress2009:1-4 [59]Shi Ruizhi Summary of grating stereoscopic printing technology[j]chinaprintingandpackagingstudy20091 (5):1-9 (inchinese) ( [J] 20091(5):1-9) [60]FlackJHarroldJWoodgateGJA prototype3d mobile phone equipped with a next-generation autostereoscopic display [C]?ProceedingsofSPIEBelingham:Society of Photo-OpticalInstrumentationEngineersPress20076490: 64900M