基于Hadoop的分布式视频处理  

Distributed Video Processing Based on Hadoop

作　　者：卞涛[1,2] 罗泽[2] 马永征[3] 

机构地区：[1]中国科学院大学,北京100049 [2]中国科学院计算机网络信息中心,北京100190 [3]中国互联网络信息中心,北京100190

出　　处：《科研信息化技术与应用》2016年第4期61-69,共9页E-science Technology & Application

基　　金：国家自然科学基金(61361126011)

摘　　要：中国科学院计算机网络信息中心与青海湖保护区管理局合作,共同建设了青海湖野外网络视频监控。如何高效地处理每天产出的超过100GB的视频数据成为了一个难题。现在的视频处理系统采用计算和存储相分离的架构,这需要配置较高的专门服务器进行支撑,本文基于廉价扩展性能较好的Hadoop平台对视频处理做出了分布式的实现,并对单个视频文件做出了分布式转码的实现。同时,本文将基于Hadoop的分布式视频处理的实现和基于HTCondor的分布式视频的批处理实现进行了对比,实验证明,在不损失视频处理效率的条件下,基于Hadoop的分布式视频处理的实现拥有分布式文件系统支撑、完善的任务监控等优势。The Computer Network Information Center of the Chinese Academy of Sciences, in cooperation with the Qinghai Lake Nature Reserve Administration, has built a Qinghai Lake outdoor video surveillance network. How to efficiently handle the daily output of more than 100 GB of video data has become an issue. The present video processing system adopts the architecture of computing and storage separation,which needs the support of highly specialized servers. This paper proposes a distributed implementation of video processing based on the Hadoop platform with better scalability and low cost. In this paper, the implementation of the distributed video transcoding is presented. At the same time, the paper compares the Hadoop-based implementation with the HTCondor-based batch implementation of distributed video. The experiment shows that the distributed video processing system based on Hadoop has the advantages of distributed file system support and perfect task monitoring without losing the efficiency of video processing.

关 键 词：视频处理 HADOOP 分布式 HTCondor HDFS 

分 类 号：TP316[自动化与计算机技术—计算机软件与理论]