针对Hadoop异构集群节点性能的数据分配策略  

Node-capability-aimed Data Distribution Strategy in Heterogeneous Hadoop Cluster

作　　者：林常航 郭文忠[1,2] 陈煌宁 

机构地区：[1]福州大学数学与计算机科学学院,福州350108 [2]国防科学技术大学计算机学院,长沙410073

出　　处：《小型微型计算机系统》2015年第1期83-88,共6页Journal of Chinese Computer Systems

基　　金：国家自然科学基金项目(61103175)资助;教育部科学技术研究重点项目(212086)资助;福建省科技创新平台项目(2009J1007)资助;福建省教育厅科技重点项目(JA11011)资助;福建省高校杰出青年人才计划项目(JA12016)资助

摘　　要：当前Hadoop的实现主要针对同构集群,假设任务处理的数据基本是本地的.然而,实际应用中集群多为异构.这暴露出现有的数据分配策略对数据局部性考虑的不足,其产生的不必要数据传输耗费了大量的带宽资源和传输时间.通过结合Hadoop中数据放置与任务执行的关系,按不同节点对不同任务的执行能力进行数据分配.在考虑异构集群中节点固有性能的情况下,提出一种机架间基于任务特性和节点计算能力的数据分配策略.该分配策略提高了对数据局部性的关注,使每个节点都尽可能只访问本地数据.通过实验可知,该策略可以有效地缩短作业执行时间,提高时效性;同时提高数据局部性,减少网络数据传输,避免拥塞;最后,该分配策略还具有较好的稳定性.The current implementation of Hadoop focuses on homogeneous cluster, and assumes that most tasks can obtain data locally. However, most clusters are heterogeneous in practical. This exposes the fact that the existing data allocation strategy does not fully con- sider data locality, which leads to unnecessary data transmission which will occupy extra network bandwidth and cost time for trans- mission. By combining the relation between data placement and task execution in Hadoop, this paper carries out data distribution ac- cording to the different nodes＇execution ability for different tasks. Considering the inherent computing property of heterogeneous clus- ter, we propose a data distribution strategy among racks based on task characteristics and node computing capacity. The allocation strat- egy pays more attention to data locality, and maximizes accesses to local data as much as possible. The experimental results demon- strate that, the allocation strategy can effectively reduce the execution time, and improve the timeliness;meanwhile, it can improve the data locality, reduce network data transmission and avoid congestion;finally, the allocation strategy also exhibits good stability.

关 键 词：HADOOP 数据分配 数据局部性 异构集群 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]