异构资源环境下的MapReduce性能优化  

Optimization of MapReduce Performance in Heterogeneous Resource Environments

作　　者：陶永才[1] 石磊[1] 

机构地区：[1]郑州大学信息工程学院,郑州450001

出　　处：《小型微型计算机系统》2013年第2期287-292,共6页Journal of Chinese Computer Systems

基　　金：国家"九七三"重点基础研究发展计划项目(2007CB310900)资助;国家"八六三"高技术研究发展计划项目(2006AA01A115)资助

摘　　要：针对现有Hadoop难以适应异构资源环境的不足,提出一种自适应MapReduce调度器:CloudMR.基于数据局部性,CloudMR将同一机架内的<key,value>对进行本地归约合并,减少中间结果中<key,value>对的数目,从而减少机架间的数据传送.根据资源性能和任务特征,CloudMR动态确定节点任务槽数和数据分配量.对于计算性能高的节点,CloudMR分配较多的任务和数据量,而对于计算性能低的节点,相应地减轻任务和数据量负载.实验表明,在异构环境下,较之现有Hadoop,Cloud-MR减少了节点间数据传输和备份任务运行,缩短了作业完成时间.An adaptive MapReduce scheduler, CloudMR, is presented and aimed at the current Hadoop model deficiency to adapt to heterogeneous resource environments. Based on data locality, CloudMR starts the local reduce tasks after all the map tasks on the same rack are successfully executed to reduce the amount of intermediate data transfer across racks. CloudMR can adaptively deter- mine the number of slot and the amount of data placement for each node in light of node performance and job character. CloudMR as- signs more tasks and data placement for high - performance nodes and little for low - performance nodes. The experiment results show that in heterogeneous resource environments, compared with current Hadoop, CloudMR reduces the amount of data transfer, the spec- ulative tasks, and degrades the job finish time.

关 键 词：云计算 异构资源 MAPREDUCE HADOOP 

分 类 号：TP309[自动化与计算机技术—计算机系统结构]