基于分布式索引和目录聚合的海量小文件存储研究  

Research on enormous storage for small files based on distributed indexing and directory aggregation

作　　者：马灿[1,2] 孟丹[1] 熊劲[1] 

机构地区：[1]中国科学院计算技术研究所高性能计算机研究开发中心,北京100190 [2]中国科学院研究生院,北京100049

出　　处：《高技术通讯》2012年第10期1035-1040,共6页Chinese High Technology Letters

基　　金：863计划（2009AA01A129）资助项目.

摘　　要：针对海量小文件访问问题的挑战，提出了用基于分布式索引和目录聚合的分布式文件系统——超虚拟文件系统（HVFS）来管理数十亿个小文件的方法，以支持高并发、高吞吐、低延迟的访问。重点讨论了目录索引、目录存储问题，提出了利用可扩展哈希索引来降低延迟、提高扩展性，利用日志结构和列存储的目录聚合来提高吞吐率的方法。测试结果表明，HVFS的存储性能能够线性扩展，82节点上峰值创建速度接近百万每秒，与GIGA＋相比有200％以上的提升，小文件I／O性能与OrangeFS相比有60倍以上的提升，充分验证了分布式索引和目录聚合方法的有效性。To solve the problem of access to a vast amount of small files created in Web service, a method for managing billions of small flies using the hyper virtual file system （HVFS）, a distributed file system based on distributed indexing and directory aggregation, is proposed to provide both high throughput and low latency file access. The measures of using distributed extendible hash indexing to improve system scalability, using log structure format and columnar storage to exploit temporal and spatial locality in directory aggregation, are also presented. The evaluation indicates that the HVFS can scale linearly and obtain 1 to GIGA ＋ , it can improve more than 200%. Compared million file creations per second with 82 nodes. to the small file I/O performance of OrangeFS, Compared ii can improve more than 6000%.

关 键 词：小文件 海量存储 分布式索引 目录聚合 分布式文件系统 

分 类 号：TP311.131[自动化与计算机技术—计算机软件与理论]