基于EHDFS的海量小文件存储与检索方法  

Storage and retrieval method of massive small files based on EHDFS

作　　者：李文武 张建锋[1] 王景林 LI Wen-wu;ZHANG Jian-feng;WANG Jing-lin(School of Information Engineering,Northwest A&F University,Yangling 712100,China)

机构地区：[1]西北农林科技大学信息工程学院,陕西杨凌712100

出　　处：《计算机工程与设计》2022年第2期376-383,共8页Computer Engineering and Design

基　　金：陕西省重点项目研发计划基金项目(2019 NY-164)。

摘　　要：为有效解决HDFS面对多类型的海量小文件存在存储效率与检索速率低下的问题,构建一种基于EHDFS架构的存取方案。存储阶段,引入最优化策略,建立新的合并存储模型,使小文件最大化填满且均匀分布于Block,提高DataNode空间利用,降低NameNode内存开销。检索阶段,改进MapFile映射关系结构、索引存储位置与组成元素以建立新的文件索引模型,避免跨跃式文件搜索,实现小文件的集中检索。实验结果表明,对比多种大数据存储模型,在不同数据量的分组压力测试下,该方案有效提高了HDFS的存取效率。To effectively solve the problems of low storage efficiency and retrieval speed of HDFS in the face of multi-type massive small files,an improved access scheme based on EHDFS architecture was constructed.In the storage stage,a merge storage model based on the optimization strategy was established to maximize the file filling that even distributed on the block,the DataNode space utilization was improved,and the NameNode memory consumption was reduced.In the retrieval stage,by improving the MapFile mapping structure,index storage location and components,a file index model was established to avoid skipping file searches and realize centralized retrieval of small files.Experimental results show that the scheme effectively improves the access efficiency of HDFS under the pressure of grouping test with different data volumes,compared with various big data storage models.

关 键 词：海量小文件 EHDFS架构 最优化合并存储模型 MapFile映射关系结构 文件索引模型 

分 类 号：TP302[自动化与计算机技术—计算机系统结构]