HDFS下载效率的优化  

Improving downloading performance in hadoop distributed file system

作　　者：曹宁[1] 吴中海[1] 刘宏志[1] 张齐勋[1] 

机构地区：[1]北京大学软件与微电子学院,北京102600

出　　处：《计算机应用》2010年第8期2060-2065,2240,共7页journal of Computer Applications

摘　　要：针对HDFS的内部数据下载效率较低和可能出现的负载不均衡的问题进行了研究,从分布式文件整体下载效率和数据块的下载效率两方面提出了优化方法。实验结果表明:两个方法都能提高效率,但在集群有大量DataNode的前提下,两者结合起来的方法能更好地提高下载效率和均衡DataNode的负载。Concerning the problems such as low downloading efficiency and imbalanced load of DataNode in Hadoop Distributed File System HDFS.Inthis paper two methods to optimize were proposed one was to improve the whole process of downloading a file the other was to optimize the downloading a block by a parallel download algorithm for dynamically allocating load by speed.Mathematical analysis and experiments prove that two methods can enhance the efficiency.Meanwhile by combining the two methods downloading is more efficient and more stable when the load of DataNode can be balanced to some extent.

关 键 词：云计算 Hadoop档案系统(HDFS) 多线程 并行下载 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构] TP393[自动化与计算机技术—计算机科学与技术]