一种优化MapReduce系统能耗的任务分发算法  

A Task Distribution Algorithm for Energy Consumption Optimization of MapReduce System

作　　者：宋杰[1] 徐澍[1] 郭朝鹏 鲍玉斌[2] 于戈[2] 

机构地区：[1]东北大学软件学院,沈阳110819 [2]东北大学信息科学与工程学院,沈阳110819

出　　处：《计算机学报》2016年第2期323-338,共16页Chinese Journal of Computers

基　　金：国家自然科学基金(61433008;61202088;61272179;61173028);教育部博士点基金(20120042110028);教育部-英特尔信息技术专项科研基金(MOE-INTEL-2012-06);中央高校基本科研业务费专项资金(N130417001);中国博士后科学基金面上项目(2013M540232);辽宁省博士启动基金(201403314)资助~~

摘　　要：MapReduce是一种典型的分布式计算模型,一经提出就被迅速应用到大数据处理系统中.文中认为MapReduce系统在能耗方面存在优化空间.对于一个分布式并行计算系统,任务的并行性对任务执行性能影响显著,并行性保证方法在优化性能的前提下还应该考虑系统能耗.在MapReduce系统中,传统的Map任务分发算法采用"小任务多次分发的策略",这种策略虽然保证了并行性,但会浪费节点的处理能力,消耗额外的能量;而Reduce任务分发算法尚不能保证Reduce任务间的并行性.文中提出通过动态地调整Map任务和Reduce任务大小,也即任务处理数据量的规模来保证任务并行性,降低MapReduce系统的整体能耗.文中通过实验证明该方法能够有效地降低典型MapReduce作业的能耗.While MapReduce is proposed as atyptcaL u huge repercussion and applied rapidly to big data processing. However, its energy consumption still can be optimized, for the distributed parallel computing system, the parallelism of tasks is the key to performance, the parallelism ensuring approach should consider not only time consumption but also energy consumption. In order to improve the parallelism, the traditional Map task distribution algorithm use the ＂fine granular task distribution strategy＂ to improve the parallelism, but it wastes the energy; and Reduce task distribution algorithm cannot guarantee the parallelism among the Reduce tasks. In this paper, we optimize MapReduce by adjusting the size of Map tasks and Reduce tasks dynamically, which can save energy consumed by MapReduce system. The algorithm proposed to this paper has been proved effective in reducing the energy consumption through a series of experiments.

关 键 词：MAPREDUCE 能耗 能耗优化 任务分发 并行性 云计算 大数据 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]