航空货运中海量小文件的存储优化  

Storage optimization of massive small file in air cargo

作　　者：李国[1] 李汶晓 徐俊洁 LI Guo;LI Wen-xiao;XU Jun-jie(School of Computer Science and Technology, Civil Aviation University of China, Tianjin 300300, Chin)

机构地区：[1]中国民航大学计算机科学与技术学院,天津300300

出　　处：《计算机工程与设计》2018年第5期1484-1489,共6页Computer Engineering and Design

基　　金：中国民航大学中央高校基金项目(31220106A001);民航科技重大专项基金项目(MHRD20140106;MHRD20160109)

摘　　要：针对Hadoop分布式文件系统在航空货运中海量小文件存储方面存在的低效率问题,提出一种面向航空货运中海量小文件的存储优化方法。通过分析航空货运数据交换平台中历史报文数据,构建用户的报文类型访问模型,计算报文类型的关联度,根据关联度将小文件合并成大文件后存储至HDFS;建立小文件索引以及数据缓存和相关小文件预取机制。实验结果表明,该方法对于Hadoop分布式文件系统存取航空货运海量小文件时NameNode的压力有明显缓解,提升了存取文件的时间效率,为航空货运中海量小文件的存储问题提供了技术支持。Focusing on the low efficiency of Hadoop distributed file system on storing lots of small files in air cargo,an optimized approach of storing lots of small files in air cargo was proposed.An accessing model of user message type was built by analyzing the historical message data in cargo data exchange platform,the correlation degree was calculated among different message types,and small files were merged into large files using the correlation degrees to store them to HDFS.The index of small files,data cache and prefetching mechanism of related small files were established.Experimental results show that the proposed approach significantly relieves the pressure of NameNode of Hadoop distributed file system to write and read small files in air cargo,enhances the time efficiency of writing and reading files,and provides considerable technical support for storing small files in air cargo.

关 键 词：航空货运数据交换平台 HADOOP分布式文件系统 海量小文件 网页排序算法 预取缓存 

分 类 号：TP302[自动化与计算机技术—计算机系统结构]