MapReduce集群中最大收益问题的研究  

Research on Maximum Benefit Problem in a MapReduce Cluster

作　　者：王习特[1] 申德荣[1] 于戈[1] 白梅[1] 聂铁铮[1] 寇月[1] 

机构地区：[1]东北大学信息科学与工程学院,沈阳110004

出　　处：《计算机学报》2015年第1期109-121,共13页Chinese Journal of Computers

基　　金：国家"九七三"重点基础研究发展规划项目基金(2012CB316201);国家自然科学基金面上项目(61033007);教育部博士点基金(20120042110028);教育部-英特尔信息技术专项科研基金(MOE-INTEL2012-06)资助~~

摘　　要：MapReduce是目前最为流行的用于大数据分析的并行系统之一.许多企业已经搭建了自己的MapReduce集群,为广大用户提供计算服务.用户可以向集群提交具有完成时限要求的MapReduce作业,若作业被按时完成,则企业可以获得一定的收益.针对这种应用场景,该文首次提出了MapReduce集群中的最大收益问题.为有效地解决该问题,首先提出了一种基于序列的任务调度策略(简称为SEQ策略),并证明了在处理具有完成时限约束的作业时SEQ策略存在优势.基于SEQ策略,该文提出了最大收益的调度算法(Scheduling Algorithm for Maximum Benefit,简称AMB算法),该算法可以快速地确定可接收作业,并给出有效的执行方案,以达到最大化收益的目的.另外,针对在实际应用中的某些异常情况(如节点宕机),该文也设计了有效的超时处理策略,进一步增加了算法的实用性.最后,通过大量的实验验证了该文所提出算法的有效性.MapReduce is one of the most popular parallel systems for big-data analysis.Many companies have built their MapReduce clusters to provide computing services to users.Users can submit their deadline-constraint MapReduce jobs to the cluster.If the jobs are finished before their deadlines,the company can get some benefits.For this application scenario,the maximum benefit problem in a MapReduce cluster is firstly presented in this paper.To solve this problem effectively,a sequence-based task scheduling strategy（SEQ strategy for short）is proposed,and we prove the advantages of SEQ strategy for the deadline-constraint job processing.Based on SEQ strategy,a novel Algorithm for Maximum Benefit,AMB,is proposed.AMB can efficiently determine the acceptable jobs and provide the effective execution strategy which can maximize the benefit.Besides,for the exceptions（e.g.node failure）in practical applications,a timeouthandling method is proposed,which can further improve the practicality of the algorithm.At last,the effectiveness of the proposed algorithm is verified through plenty of experiments.

关 键 词：大数据 MapReduce集群 完成时限 最大收益问题 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]