Hadoop任务分配策略的改进  

Tasks assignment optimization in Hadoop

作　　者：黄承真[1] 王雷[1] 刘小龙[1] 况亚萍[1] 

机构地区：[1]中国科学技术大学自动化系,合肥230027

出　　处：《计算机应用》2013年第8期2158-2162,共5页journal of Computer Applications

基　　金：中央高校基本科研业务费专项资金资助项目(WK2100100012)

摘　　要：Hadoop广泛应用于大数据的并行处理,其现有的任务分配策略多面向同构环境,或者没有充分利用集群的全局信息,或者在异构环境下无法兼顾执行效率与算法复杂度。针对这些问题,提出异构环境下的任务分配算法λ-Flow算法,将原先一次完成的任务分配过程划分成多轮,每轮基于当前集群状态,以及上轮任务的执行情况,动态进行任务分配,直至全部任务分配结束,以期达到最优执行效率。通过与其他算法对比实验表明,λ-Flow算法能够更好地适应集群的动态变化,有效减少作业执行时间。Hadoop has been widely used in large data parallel processing.The existing tasks assignment strategies are almost oriented to a homogenous environment,but ignore the global cluster state,or not take into account the efficiency of the implementation and the complexity of the algorithm in a heterogeneous environment.To solve these problems,a new tasks assignment algorithm named λ-Flow which was oriented to a heterogeneous environment was proposed.In λ-Flow,the tasks assignment was divided into several rounds.In each round,λ-Flow collected the cluster states and the execution result of the last round dynamically,and assigned tasks in accordance with these states and the result.The comparative experimental result shows that the λ-Flow algorithm performs better in a dynamic changing cluster than the existing algorithms,and reduces the execution time of a job effectively.

关 键 词：HADOOP MAPREDUCE 任务分配 异构环境 最小费用最大流 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构] TP393.027.2[自动化与计算机技术—计算机科学与技术]