面向搜索引擎的分布式文件系统性能分析  

Performance Analysis of Distributed File System for Search Engine

作　　者：董守斌[1] 赵铁柱[1] 

机构地区：[1]华南理工大学广东省计算机网络重点实验室,广东广州510640

出　　处：《华南理工大学学报（自然科学版）》2011年第4期7-14,共8页Journal of South China University of Technology(Natural Science Edition)

基　　金：国家自然科学基金资助项目(61070092);国家发改委CNGI项目(CNGI2008-109/122)

摘　　要：搜索引擎是一种数据密集型应用,其性能极大依赖于底层文件系统的性能.文中主要讨论分布式文件系统在搜索引擎应用环境下的性能评估和性能优化问题.首先概述了分布式文件系统的性能影响因素及相关研究进展;在此基础上提出基于Hadoop的开放架构,系统地评估HDFS和Lustre在搜索引擎应用场景下的性能;最后针对实验评估发现的HDFS在写性能及小文件数据处理方面的不足,提出改进方案,为搜索引擎的分布式文件系统优化提供参考.As a search engine is a kind of data-intensive application,its performance is greatly affected by the underlying distributed file system.This paper deals with the performance evaluation and optimization of the distributed file system oriented to search engine application.In the investigation,first,the factors affecting the performance of distributed file system and the relevant research progress are summarized.Then,an open architecture based on Hadoop is designed to systematically evaluate the performance of HDFS and Lustre in search engine scenarios.Finally,several improved schemes are proposed to overcome the shortcomings of HDFS in terms of write performance and small file disposition clarified by the results of performance assessment,which provides a reference for the optimization of distributed file system.

关 键 词：搜索引擎 分布式文件系统 HDFS文件系统 LUSTRE文件系统 性能分析 性能优化 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]