一种优化的Hadoop副本放置策略  

An improved replica placement strategy in Hadoop

作　　者：蔡燕冬[1] 刘艳[1] 张庆磊[1] 

机构地区：[1]华侨大学计算机科学与技术学院,福建厦门361021

出　　处：《微型机与应用》2015年第16期21-23,共3页Microcomputer & Its Applications

基　　金：国家自然科学青年基金项目(61202106)

摘　　要：Hadoop分布式文件系统默认采用三副本策略实现较为简单,未对数据节点负载进行充分考虑。为了改善HDFS中集群负载的均衡性,提高数据节点的资源利用率,提出一种优化的副本放置策略。该策略综合考虑数据节点的实时负载信息和工作进程数,选择负载最小的节点存放数据。实验结果表明,与默认策略相比,优化的Hadoop副本放置策略能使副本分布更加合理,集群的均衡性更加良好,并能减少数据上传响应时间。Hadoop distributed file system applies default three copies of the random replica placement strategy without taking into account full load of Datanodes. To improve the cluster load balabcing of HDFS and the resource utilization of Datanodes, an improved replica placement strategy is proposed. The strategy considers real-time load of Datanodes and the number of the work process to select the minimum load Datanode storing data. Experiment shows that compared with default three copies of the random replica placement strategy, the improved strategy optimizes the balancing of cluster load and reduces I/O response time.

关 键 词：HADOOP 副本放置 实时负载 负载均衡 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术] TP911[自动化与计算机技术—计算机科学与技术]