基于Hadoop的海量车牌图像处理优化技术  

A Processing Optimization Technique for Massive License Plate Images Based on Hadoop

作　　者：侯向宁[1] HOU Xiang-ning(Engineering ＆Technical College of Chengdu University of Technology,Leshan 614007,China)

机构地区：[1]成都理工大学工程技术学院,四川乐山614007

出　　处：《计算机技术与发展》2018年第10期135-138,144,共5页Computer Technology and Development

基　　金：四川省教育自然科学重点项目(12ZA200);成都理工大学工程技术学院青年科学基金(C122016006)

摘　　要：Hadoop集群下每个小文件均占据一个Block,一方面存储海量元数据信息消耗了大量的NameNode内存,另一方面,Hadoop为每个小文件单独启动一个Map任务,大量的时间花费在启动和关闭Map任务上,从而严重降低了MapReduce的执行速率。对此,在详细分析已有解决方案的基础上,采用CFIF将多个小文件分片打包到大分片中,给每个大分片只启动一个Map任务来执行,通过减少启动Map任务的数量,提高了处理海量小文件时的效率。通过设计Hadoop图像接口类,继承并实现CFIF抽象类,最终完成了对海量图像小文件的处理。与常规HDFS、HAR和MapFile方案在NameNode内存空间和运行效率方面进行了对比,结果表明,CFIF在NameNode内存占用率和运行效率方面,都有很好的表现。Under the Hadoop cluster,each small file occupies a block. On the one hand,to store massive metadata information consumesa lot of NameNode memory;on the other hand,Hadoop starts a Map task for each small file,spending a lot of time on startup and shutdown Map tasks,which severely reduces the execution speed of the MapReduce. In view of this,on the basis of analysis of several existing solutions,we use CFIF abstract class to package multiple small files into a big split,for each big split only start a Map task to perform. By reducing the number of Map tasks,we improve the efficiency when dealing with massive small files. Through designing theHadoop image interface class,we inherit and implement CFIF abstract class for final completion of the processing of large image smallfiles. The comparison between CFIF and conventional HDFS,HAR and MapFile solutions in the NameNode memory usage rate and operating efficiency shows that the CFIF performs well.

关 键 词：海量小文件 HADOOP分布式文件系统 分片 打包 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]