untitled

Transcription

1 ISSN , CODEN RUXUEW Journal of Software,2017,28(8): [doi: /j.cnki.jos ] Tel: ,2, 1,2, 1,2, 1,2, 1,2 1 ((), ) 2 (, ) :, csyhua@hust.edu.cn :,.,, RAW,.,,,,.,., PC P, 2.5. : ;;;; : TP316 :,,,,..,2017,28(8): : Cao SD, Hua Y, Feng D, Sun YY, Zuo PF. High-Performance distributed storage system for large-scale highdefinition video data. Ruan Jian Xue Bao/Journal of Software, 2017,28(8): (in Chinese). High-Performance Distributed Storage System for Large-Scale High-Definition Video Data CAO Shun-De 1,2, HUA Yu 1,2, FENG Dan 1,2, SUN Yuan-Yuan 1,2, ZUO Peng-Fei 1,2 1 (Wuhan National Laboratory for Optoelectronics (Huazhong University of Science and Technology), Wuhan , China) 2 (School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan , China) Abstract: This paper presents a high-performance distributed storage solution for video surveillance data via analyzing the characteristics of video surveillance data and conventional file storage solutions. This design proposes a logical volume structure rather than file-based storage to efficiently organize unstructured video stream data. The scheme hence directly writes these stream data into raw disk devices, to address the problem of storage performance decrease caused by the random access and disk fragmentation in traditional storage systems. A two-stage index strategy is also implemented to manage metadata by the state manager and storage servers, which significantly reduces the amount of metadata managed by the state manager, eliminates the performance bottlenecks, and provides the second-level video retrieval accuracy. Moreover, the design has the salient features of fault tolerance and linear scaling abilities with the help of the flexible storage server grouping policy and the mutual backup relationship in a storage group. Experimental results show that the solution can simultaneously record 400 ways of 1080P video streams with a single low-cost PC server, and the system s average write speed is 2.5 times faster compared with the local file systems. Key words: massive storage; video storage; stream media storage; distributed system; logical volume : (2016YFB ) Foundation item: National Key Research and Development Program of China (2016YFB ) : ; : , ; : ; jos : CNKI : :36:01,

2 2000 Journal of Software Vol.28, No.8, August 2017,,.,. IP-SAN(Internet protocol storage area network(ip )) iscsi(internet small computer system interface(internet )),.,,. 7 24,,,,.,., [1],.,,. (1).,, PB. 30 ~90,,. (2)., 20%,80%., 100%,.. (3).,,. (4) 7 24.,.,,,., DVSS(distributed video surveillance storage).. (1), RAW,,,. (2),.,,. (3), DVSS,, DVSS.,DVSS PC P, ,,,, [2 4]. [2],.,,,.[3] (group of pictures, GOP),,[2].[5] Ripple-RAID,,.[6] THNVR.THNVR,, SQLite,,

3 : 2001.THNVR,. [7] Hadoop (HDFS), HDFS, Map/Reduce., HDFS.,[8] DSFS.DSFS (continuous storage model, CSM),. DSFS MetaServer,,,,. 2 DVSS. NTFS Ext4,,. (1),,,,.,,,,. (2) :, inode ;,. 2.1 DVSS DVSS (state manager)(storage server)(client). 1. DVSS / / / ( ) ( ) N N N Fig.1 Architecture of DVSS 1 DVSS,.,., 1,,,.HDFS

4 2002 Journal of Software Vol.28, No.8, August 2017,, 3.DVSS,.,,.,;,,.. ( SID), IP. 2.2,,DVSS.,, ,DVSS GOP,., 2. GOP #1 GOP #2 GOP #3 GOP #4 GOP # N Fig.2 Logical volume architecture design 2 (1),,,. LVM(logical volume manager()), 2. (super block),,.,0,1.,,. (2)., bit 1.,,.,,. 512MB, 3.2. (3) GOP H.264/AVC I P B, GOP 1s. DVSS, GOP, GOP, GOP.

5 : ,DVSS, Redis. (1),. 3,,, ID.(SID) Key, Value.,, O(1). Fig.3 First-Level index structure 3,. (2),, GOP, GOP. GOP GOP. Key, GOP Value.,GOP, O(1) ,,., 1.,;,.. (1),. (2),, IP. (3),. (4) GOP,, GOP

6 2004 Journal of Software Vol.28, No.8, August 2017, GOP. (5), ,,.,,.. (1),. (2) Key,,., IP. (3),,. (4),, GOP. (5). 2.4 DVSS.,.,,,,.,,,.,,,,,.,,,, ,,,.,,. DVSS..,,., 0,,.,,.,,.,, , DVSS. (1),,.,., LVM,,,(/dev/raw/

7 : 2005 raw[n]),. (2),.,,,,,,. 2.6 DVSS PC,,, 7 24.DVSS. (1) HDFS NameNode,DVSS. Hadoop 2.0,HDFS NameNode.DVSS HDFS HA [9],,.,.,,.,.,. (2).DVSS.,..,,.,,,. (3),, SD., ;, DVSS,., DVSS, DVSS 40. Intel Xeon 2.4GHz 2 CPU, 4GB,6 1TB 15000RPM SATA Disks Mellanox InfiniBand QDR 40Gb/s. Ubuntu Server 14.04LTS, Redis V ,,. 4. :64MB~512MB, 64KB, 32MB,. 64MB~600MB,100,.: blocksize, bufsize. bufsize

8 2006 Journal of Software Vol.28, No.8, August 2017, lseek(), offset blocksize bufsize. offset, write() write(),,,.,, bufsize, offset,., 600MB,.,DVSS 512MB 1MB. Fig.4 Effect of data block and buffer size on disk write performance DVSS, 50 ~ P DVSS Ext4, 4Mbit/s, 5. Fig.5 DVSS single storage server and Ext4 performance comparison evaluation 5 DVSS Ext , 25MB/s 50MB/s,Ext4 DVSS. 200, 100MB/s,DVSS, Ext MB/s,,. 300, 150MB/s,DVSS

9 : 2007, Ext MB/s., DVSS,Ext4., Ext4,,,,,.,DVSS, P. 3.4 DVSS, DVSS, , 2.(constant-bit-rate traffic generator) , 4Mbit/s. 6,DVSS , 550, 275.,DVSS,. DVSS,,,,.,. Fig.6 DVSS overall performance evaluation 6 DVSS 3.5 DVSS, DVSS, 1. Table 1 Comparison of DVSS with exisiting schemes 1 DVSS / THNVR [6] SQLite+ SATA 256 1~2Mbit/s DSFS [8] 300 CSM 4Mbit/s DVSS Redis+ 400, 4Mbit/s

10 2008 Journal of Software Vol.28, No.8, August 2017 THNVR, SQLite.THNVR,.THNVR 1, THNVR,,.DSFS (CSM),,. CSM,,, DSFS.DSFS MetaServer,,.DVSS GOP,.DVSS,,,,,.,DVSS,,. 4 IP-SAN, DVSS.DVSS GOP,, P 2.5.DVSS,,.,,., DVSS,. DVSS,, SSD(solid state disk()),.,(erasure code) 1,. References: [1] Wang YJ, Sun WD, Zhou S, Pei XQ, Li XY. Key technologies of distributed storage for cloud computing. Ruan Jian Xue Bao/ Journal of Software, 2012,23(4): (in Chinese with English abstract). [doi: /SP.J ] [2] He J. Study on key technique in video surveillance storage system based on IP-SAN [MS. Thesis]. Shanghai: Shanghai Jiaotong University, 2011 (in Chinese with English abastract). [3] Tang JX. The software design of storage subsystem for network video surveillance system [MS. Thesis]. Hangzhou: Zhejiang University, 2013 (in Chinese with English abastract). [4] Jiang M, Niu ZY, Zhang SP. Design and implementation of video surveillance storage system. Computer Engineering and Design, 2014,35(12): (in Chinese with English abstract). [doi: /j.issn ] [5] Sun ZZ, Zhang QX, Tan YA, Li YZ. Ripple-RAID: A high-performance and energy-efficient RAID for continuous data storage. Ruan Jian Xue Bao/Journal of Software, 2015,26(7): (in Chinese with English abstract). [doi: /j.cnki.jos ] [6] Wu JY, Gu Y, Ju DP, Wang DS. THNVR: Distributed large-scale surveillance video storage system. Computer Engineering and Applications, 2009,45(31):56 59 (in Chinese with English abstract). [doi: /j.issn ] [7] Lin CF, Yuan SM, Leu MC, Tsai CT. A framework for scalable cloud video recorder system in surveillance environment. In: Proc. of the Ubiquitous Intelligence & Computing and 9th Int l Conf. on Autonomic & Trusted Computing (UIC/ATC). Fukuoka: IEEE, [doi: /UIC-ATC ] [8] Duan HC, Zhan WH, Min GY, Guo H, Luo SM. A high-performance distributed file system for large-scale concurrent HD video streams. Concurrency and Computation: Practice and Experience, 2015,27(13): [doi: /cpe.3528] [9] Deng P, Li MY, He C. Research on namenode single point of fault solution. Computer Engineering, 2012,38(21):40 44 (in Chinese with English abstract). [doi: /j.issn ]

11 : 2009 : [1],,,,..,2012,23(4): /4175.htm [doi: /SP.J ] [2]. IP-SAN [].:,2011. [3].[].:,2013. [4],,..,2014,35(12): [doi: /j.issn ] [5],,,.Ripple-RAID:.,2015,26(7): jos.org.cn/ /4606.htm [doi: /j.cnki.jos ] [6],,,. THNVR.,2009,45(31): [doi: /j. issn ] [9],,.Namenode.,2012,38(21): [doi: /j.issn ] (1991),,,, (1992),,,CCF,,,.. (1978),,,,, CCF,,. (1992),,,CCF,,,. (1970),,,,, CCF,.