一种基于动态资源采集的Hadoop作业调度算法  

A Scheduler for Hadoop Based on Dynamic Resource Collection

作　　者：彭志伟[1] 谷建华[1] 

机构地区：[1]西北工业大学计算机学院,陕西西安710072

出　　处：《微电子学与计算机》2014年第4期64-67,共4页Microelectronics & Computer

摘　　要：为了克服Hadoop中基于槽的资源模型对集群资源的浪费,提高Hadoop集群的资源利用率,提出了一种基于动态资源采集的Hadoop作业调度算法.通过在作业运行过程中动态采集部分任务的CPU,内存和IO的资源利用率来评估同一作业中其他任务的资源需求,然后根据任务的实际资源需求以及TaskTracker节点的负载情况进行任务调度,以充分利用各TaskTracker节点的计算资源.通过对比实验表明该调度算法能明显提高集群的资源利用率,缩短作业的完成时间.In order to overcome the problem of wasting of resources in hadoop cluser based on the slot resource model,a scheduler based on the dynamic resource collection has been proposed. According to collect theCPU, Memory and IO resource utilization of part of tasks in the hadoop job,We estimate the resource demand of the rest tasks in the job, and then schedule these tasks according to their resource demand and the load of every Tasktracker. The contrast experiment shows the scheduler we proposed can increase the resource utilization of the hadoop cluser and reduce the completion time of the Hadoop job.

关 键 词：HADOOP MapReduce调度 资源需求 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]