Hadoop平台下的动态调度算法  

Dynamic Scheduling Algorithm in Hadoop Platform

作　　者：高燕飞[1] 陈俊杰[1] 强彦[1] 

机构地区：[1]太原理工大学计算机科学与技术学院,太原030024

出　　处：《计算机科学》2015年第9期45-49,69,共6页Computer Science

基　　金：国家自然科学基金(61202163;61240035);山西省自然科学基金(2012011015-1);山西省科技攻关项目(20120313032-3)资助

摘　　要：目前,云计算环境具有动态、异构和海量多类型任务并发等特征,随着集群规模不断增大、用户QoS不断增多,现有调度算法越来越难以适应动态变化的环境及满足用户的需求。针对Hadoop平台下现有调度器不能根据作业运行状态和资源使用情况进行动态调整的问题,提出了Hadoop下基于作业分类的动态调度算法。该算法在使用朴素贝叶斯分类算法对队列中作业进行分类的过程中,根据各个作业的类型,预先设定类别权值,将队列中的作业分类,并引入效用函数,根据用户提交时的预期完成时间QoS和作业完成情况估算其作业完成时间,实现动态设置作业优先级。实验表明,使用提出的算法不仅能有效减少作业的分类时间,而且能明显提高动态性和用户QoS。With the increasing of the clusters and the user＇s QoS in the cloud environment, it becomes much harder to meet the requirements of jobs and users using the traditional strategy. To adjust scheduler dynamically according to the status of the jobs and the resources, this paper proposed a dynamic scheduling method based on the job classification method in the Hadoop platform. The proposed method employs the Naive Bayesian method to classify the jobs in which the human inferences are added to preset the jobs＇ weight according to the types. Then, the scheduling priority of the jobs is set dynamically using the utility function based on the user＇s expected completing time and the estimated com- pleted time of jobs. The experimental results show that the proposed method can not only reduce the classification time, but also improve the scheduling dynamics and user＇s QoS greatly.

关 键 词：人机交互 HADOOP 动态调度 贝叶斯网络 QOS 

分 类 号：TP39[自动化与计算机技术—计算机应用技术]