一种基于资源预取的Hadoop作业调度算法  

Hadoop job scheduling algorithm based on resource prefetching

作　　者：万兵[1] 黄梦醒[1] 段茜[1] 

机构地区：[1]海南大学信息科学技术学院,海口570228

出　　处：《计算机应用研究》2014年第6期1639-1643,共5页Application Research of Computers

基　　金：国家自然科学基金资助项目(71161007);国家教育部社科基金资助项目(10YJCZH049);海南省重点科技计划资助项目(ZDXM20120061;ZDXM20130078)

摘　　要：Hadoop原有的作业调度算法在任务级调度阶段均采用简单考虑数据本地性的任务分配策略,并不能保证良好的数据本地性,而相应的改进算法改进目标不一,也不完全适用于所有作业调度场合。针对以上问题,结合数据预取理念提出一种基于资源预取的作业调度算法。通过估算节点上正在运行任务的剩余执行时间,并与传输一个数据块所需时间进行比较,预选出待预取的候选节点;并从当前正在运行作业的未分配任务中选取非本地map任务作为待预取任务;最后,为该任务在候选节点上预取其所需数据。实验结果表明该改进算法有效地提高了作业的数据本地性,并在一定程度上减少了作业执行时间。The existing Hadoop job scheduling algorithms can＇ t guarantee good data locality by using a simple task allocation policy in task-level scheduling phase,and the corresponding improved algorithms vary widely in optimization goal,which are inapplicable to all Hadoop job scheduling situations.To address problems above,combining with the concept of data prefetching,this paper proposed a job scheduling algorithm based on resource prefetching.First of all,by comparing the estimated remaining execution time of running map tasks on all TaskTrackers with the time of transmitting an input data block between TaskTrackers,the algorithm selected a candidate TaskTracker to accomplish the prefetching operation.Secondly,it selected a non-local map task from all unassigned map tasks of the current running job to prefetch.Finally,it prefetched the required input data block of this candidate map task into the candidate TaskTracker＇ s disk.The experimental results indicate that the proposed algorithm improves data locality effectively and to some extent reduces the job execution time.

关 键 词：作业调度 数据本地性 资源预取 HADOOP MAPREDUCE 云计算 

分 类 号：TP393[自动化与计算机技术—计算机应用技术] TP301.6[自动化与计算机技术—计算机科学与技术]