HDFS中海量小文件合并与预取优化方法的研究  

Research on Optimization Method of Merging and Prefetching for Massive Small Files in HDFS

作　　者：郑通 郭卫斌[1] 范贵生[1] 

机构地区：[1]华东理工大学信息科学与工程学院,上海200237

出　　处：《计算机科学》2017年第B11期516-519,541,共5页Computer Science

基　　金：国家自然科学基金资助项目(61672227)资助

摘　　要：HDFS在存储海量文件时具有明显的优势,但在存储小文件占绝大多数的海量文件时,HDFS单个NameNode的存储架构会导致其性能严重降低。为此,提出一种基于合并思想的方案,即将小文件合并为大文件,同时建立小文件到合并文件的映射关系,并将其存于HBase中。为了提高读取速度,建立了基于LRU的预取机制。实验表明,该方法能明显提高HDFS在处理海量文件时的整体性能。HDFS has a significant advantage on storing the massive files,however,its storage architecture which has only one NameNode will result in the decrease of performance when HDFS is used to store massive files which is mainly composed by small files.A solution based on the idea of that small files were merged into large files was proposed.Meanwhile,the mapping relationship from small files to merging files was established and stored into HBase.Finally,we provided a LRU based prefetching mechanism to improve the reading speed.The experiments show that the proposed method can improve the overall performance of HDFS with the large amounts of small files.

关 键 词：HDFS 海量文件 合并 映射 LRU 预取机制 

分 类 号：TP302.1[自动化与计算机技术—计算机系统结构]