一种异构环境下的基于MapReduce任务调度改进机制  

Improved task scheduling mechanism for MapReduce in heterogeneous environment

作　　者：何翔[1] 李仁发[1,2] 唐卓[1,2] 

机构地区：[1]湖南大学嵌入式系统及网络实验室,长沙410082 [2]湖南大学网络与信息安全湖南省重点实验室,长沙410082

出　　处：《计算机应用研究》2013年第11期3370-3373,3379,共5页Application Research of Computers

基　　金：国家自然科学基金资助项目(60673061;60873074)

摘　　要：针对在异构环境下采用现有MapReduce任务调度机制可能出现各计算节点间数据迁移和系统资源分配难以管理的问题,提出一种动态的任务调度机制来改善这些问题。该机制先根据节点的计算能力按比例放置数据,然后通过资源预测方法估计异构环境下MapReduce任务的完成时间,并根据完成时间计算任务所需的资源。实验结果表明,该机制提高了异构环境下任务的数据本地性比例,且能动态地调整资源分配,以保证任务在规定时间内完成,是一种有效可行的任务调度机制。The existing MapReduce scheduling mechanism that used in heterogeneous environment may lead to the migration of data between compute nodes, and manage system resource allocation difficulty. This paper proposed a dynamic task scheduling mechanism to improve these problems. First, this mechanism distributed data in proportion according to the computing capacity of each node. Then it estimated MapReduce job completion time in heterogeneous environment by using resource prediction model, and calculated task slot requirements based on its completion time. The experiment results show that this mechanism can improve the proportion of task data locality, and dynamically allocate resources in order to ensure the job is completed in the the specified time. It is demonstrated that this task scheduling mechanism is effective and feasible.

关 键 词：MAPREDUCE 调度算法 资源预测 数据放置 异构环境 

分 类 号：TP302[自动化与计算机技术—计算机系统结构]