大数据应用中节点休眠结合MapReduce作业的能量感知调度方法  

A ENERGY-AWARE SCHEDULING METHOD OF NODE DORMANCY COMBINED WITH MAPREDUCE JOBS IN BIG DATA APPLICATION

作　　者：邵孟良[1] 齐德昱[2] Shao Mengliang;Qi Deyu(School of Information Engineering,Guangzhou Nanyang Polytechnic College,Guangzhou 510925,Guangdong,China;School of Computer Science and Technology,South China University of Technology,Guangzhou 510006,Guangdong,China)

机构地区：[1]广州南洋理工职业学院信息工程学院,广东广州510925 [2]华南理工大学计算机科学与工程学院,广东广州510006

出　　处：《计算机应用与软件》2020年第6期40-47,82,共9页Computer Applications and Software

基　　金：国家自然科学基金项目(61070015);广东省自然科学基金团队项目(10351806001000000);广东省前沿与关键技术创新项目(2014B010110004);广州市科技计划项目(201804010402);广州市产学研协同创新重大项目(201604016074);广州南洋理工职业学院创新强校工程项目(NY-2018CQPT-01);南洋产业研究院基金项目(NYCYYJ2016009)。

摘　　要：针对大数据应用中,数据中心执行大规模数据密集型应用程序时能耗巨大的问题,提出一种节点休眠结合MapReduce作业的能量感知调度方法。构建一个能够同时提高MapReduce应用程序能效并满足服务水平协议(service level agreement,SLA)的系统框架,用于执行大规模数据处理;将节点休眠调度算法与单个Map-Reduce作业的能量感知调度问题建模为整数程序;结合两种能量感知的节点休眠MapReduce启发式调度算法(Node Dormancy MapReduce Scheduling Algorithms,NDMRSA-1和NDMRSA-2),以实现任务分配的自动映射和任务到机器插槽的自动归并,从而最大限度地减少执行应用程序时消耗的能量。实验结果表明,NDMRSA-1和NDMRSA-2能够找到接近最佳的工作调度,平均消耗的能量比通过最小化完工时间的通用实践调度程序少约40%,有效实现了作业时的能耗最小化。Aiming at the problem of huge energy consumption when data centers execute large-scale data-intensive applications in big data applications,we propose an energy-aware scheduling method of node dormancy combined with MapReduce jobs.We built a system framework that could improve the energy efficiency of MapReduce applications and meet the service level agreement(SLA)at the same time,which was used to perform large-scale data processing.Then,node dormancy scheduling algorithm and the energy-aware scheduling problem of a single MapReduce job were modeled as an integer program.Finally,we combined two energy-aware node dormancy MapReduce scheduling algorithms(NDMRSA-1 and NDMRSA-2)to achieve automatic mapping of task assignments and automatic merging of tasks into machine slots,minimizing the energy consumption when executing the application.The experimental results show that NDMRSA-1 and NDMRSA-2 can find the best job scheduling,and the average energy consumed is about 40%less than that obtained by the general practical scheduler to minimize the completion time,which effectively realizes the energy consumption minimization during the job.

关 键 词：MAPREDUCE 节点休眠 大数据 能量感知调度 最小能量消耗 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]