基于机器学习的MapReduce资源调度算法  

Resource scheduling algorithm for MapReduce based on machine learning

作　　者：于倩[1] 蔚承建[1] 王开[2] 朱林军[1] 

机构地区：[1]南京工业大学电子与信息工程学院,南京210009 [2]东南大学信息科学与工程学院,南京210018

出　　处：《计算机应用研究》2016年第1期111-114,共4页Application Research of Computers

基　　金：江苏省自然科学基金资助项目(BK2011060;BK2010240)

摘　　要：针对MapReduce中允许map和shuffle阶段重叠的优化模型需要自适应性的问题,提出了基于此模型的机器学习的资源调度算法,利用贝叶斯分类器依据作业对系统资源的需求和系统环境的匹配程度对作业进行调度,并不断更新分类器,使其具有自适应性,考虑了map和shuffle的重叠阶段。通过模拟实验验证,改进后的算法能够提高MapReduce系统的性能,获得更好的平均响应时间。To solve the problem that a map and shuffle optimization model allowed overlapping phases in MapReduce was lack of adaptability, this paper proposed a resource scheduling algorithm based on this model of machine learning, it used Bayesian classifier based on the degree of matching the operating system resource requirements and system environment for job schedu- ling, and then updated the classification continuously, so that it had adaptability, considering the overlap phase map and shuf- fle. The simulation experiments verify that the solution can improve the performance of MapReduce system, get a better ave- rage response time.

关 键 词：MAPREDUCE 重叠阶段 自适应性 机器学习 贝叶斯分类器 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程] TP301.6[自动化与计算机技术—控制科学与工程]