Hadoop平台下新型图像并行处理模型设计  

New Design of Image Parallel Processing Model Based on Hadoop Platform

作　　者：刘军[1] 李威 吴梦婷 陈起凤 LIU Jun;LI Wei;WU Mengting;CHEN Qifeng(School of Computer Science and Engineering,Wuhan Institute of Technology,Wuhan 430205,China)

机构地区：[1]武汉工程大学计算机科学与工程学院,武汉430205

出　　处：《计算机工程与应用》2019年第6期186-190,共5页Computer Engineering and Applications

基　　金：湖北省智能机器人重点实验室开放基金(No.HBIR 201608);武汉工程大学研究生创新基金(No.CX2016063)

摘　　要：Hadoop在处理海量小图像数据时,存在输入分片过多以及海量小图像存储问题。针对这些问题,不同于采用HIPI、SequenceFile等方法,提出了一个新型图像并行处理模型。利用Hadoop适合处理纯文本数据的特性,本模型使用存储了图像路径的文本文件替换图像数据作为输入,不需要设计图像数据类型。在Map阶段直接完成图像的读取、处理、存储过程。为了简化图像处理算法,将OpenCV和Map函数结合并设计了对应的存储方法,实现小图像文件的存储。实验表明,在Hadoop分布式系统平台下,模型不论在小数据量还是在大数据量的测试数据环境中,都具有良好的吞吐性能和稳定性。While dealing with huge amount of small image data,Hadoop has the problems of managing the excessive fragmentation of the inputs and saving the rapid growth of small image files.In view of solving these problems,the solution of a new mass small image parallel processing model is proposed and implemented,and is different from the methods such as HIPI and SequenceFile.For Hadoop is suitable for the text-only data processing,the image data is replaced by the text file that stores the image path as input,and the model does not need to design image data types.The functions such as image reading,image processing,image storage are completed in the Map stage of Hadoop.And to simplify the image processing algorithms,the OpenCV functions are combined with the Map function and the corresponding storage method is designed to accommodate the storage of small image files.Experimental results show that,the model has good performance on throughput test and good stability wherever the test data is the small amount of data or large amount of data in Apache Hadoop system.

关 键 词：HADOOP 并行计算框架(MapReduce) 图像处理 OPENCV 

分 类 号：TP338.8[自动化与计算机技术—计算机系统结构]