HDAS:异构集群上Hadoop+框架中的动态亲和性调度  

HDAS: heterogeneous dynamic affinity scheduling in Hadoop +

作　　者：何文婷[1,2,3] 崔慧敏[1,2] 冯晓兵[1,2] 

机构地区：[1]中国科学院计算机体系结构国家重点实验室,北京100190 [2]中国科学院计算技术研究所,北京100190 [3]中国科学院大学,北京100049

出　　处：《高技术通讯》2016年第4期333-343,共11页Chinese High Technology Letters

基　　金：973计划(2011CB302504);863计划(2012AA010902;2015AA011505);国家自然科学基金(61202055;61221062;61303053;61432016;61402445)资助项目

摘　　要：针对现有异构集群的编程框架着重于异构资源的利用,没有充分考虑共享资源竞争导致作业完成时间延长的情况,基于Hadoop+框架和异构任务模型,提出并实现了异构动态亲和性调度(HDAS)算法,该算法利用Hadoop的心跳机制监测各结点上的资源使用情况和实时负载,对系统中的异构资源用不同的策略计算与任务的亲和性,进行任务分派,使系统的资源利用更充分,从而降低共享资源竞争导致的任务延迟,提高系统的整体吞吐率,且提交到系统中的应用都会在启动后一定时间内被执行。对25种混合负载的试验表明,Hadoop+框架使用HDAS相对于Hadoop的实现可获得平均21.9x的加速比,明显优于基于异构任务模型的调度策略(17.9x),并使其中21个负载的任务平均延迟不超过6%,在任务对系统资源需求多样性丰富的混合负载上优化效果明显。Aiming at the problem that the existing programming frameworks for heterogeneous clusters focus on utilizing heterogeneous resources in clusters while ignoring the delay of the working time caused by the contention of shared resources, this study proposed and implemented the heterogeneous dynamic affinity scheduling （HDAS） algorithm based on the Hadoop ＋ and the heterogeneity model, which dynamically calculates the real-time resource affinity of each available task according to the corresponding strategy of each resource and dispatches the most appropriate task launch in the system within a reasonable time to minimize the shared resource contention among simultaneously run- ning tasks to improve the overall throughput of the system. The experimental result showed an average 21.9x overall speedup of the HDAS compared to the Hadoop implementation on 25 hybrid workloads, while the original heteroge- neous model based scheduling strategy in Hadoop only obtained 17.9x speedup. And it showed obvious improve- ment on the hybrid workloads consisting of the tasks with more diverse resource requirements, and 21 of the 25 workloads presented the task delay of less than 1.06x in average.

关 键 词：MAPREDUCE 异构 Hadoop+ 亲和性 调度 

分 类 号：TP332[自动化与计算机技术—计算机系统结构] TP301.6[自动化与计算机技术—计算机科学与技术]