Hadoop集群中给定候选任务集的最大利润问题  

Maximum Profit Problem of a Given Candidate Set in Hadoop Cluster

作　　者：郑羽 胡积宝 ZHENG Yu;HU Ji-bao(Modern Education and Technology Center,Anqing Normal University,Anqing 246011,China;School of Physics and Electronic Engineering,Anqing Normal University,Anqing 246133,China)

机构地区：[1]安庆师范大学现代教育技术中心,安徽安庆246011 [2]安庆师范大学物理与电气工程学院,安徽安庆246133

出　　处：《计算机技术与发展》2018年第12期194-199,共6页Computer Technology and Development

基　　金：安徽省2016年高校自然科学基金重点项目(KJ2016A430)

摘　　要：随着计算机网络和传感器网络的迅速发展,数据呈指数级增长,特别是在因特网上。为了有效地处理大规模数据,需要具有良好的可伸缩性、灵活性和容错性的并行分布式集群。目前,许多企业基于自己的Hadoop集群提供云服务。因为单个Hadoop集群的资源是有限的,Hadoop集群必须将有限的资源分配给一些特殊的任务以获得最大的利益。文中研究给定候选任务集的最大利润问题。用有效的序列描述候选任务集,并提出了一种基于序列的调度策略。为了提高查找有效序列的效率,设计了一些修剪策略,并给出了相应的调度算法。最后,在某些任务运行超时的情况下,提出了超时处理算法。实验结果表明,该算法的总收益非常接近理想的最大值,在不同的实验环境下明显优于相关的调度算法。With the rapid development of computer networks and sensor networks,data are exponentially increased,especially on the Internet.In order to deal with large-scale data efficiently,a parallel and distributed cluster with better scalability,flexibility and fault tolerance is needed.Nowadays,many enterprises provide cloud services based on their own Hadoop clusters.Because the resources of a Hadoop cluster are limited,the Hadoop cluster must select some specific tasks to allocate limited resources in order to get the maximal profit.In this paper,we study the maximal profit problem for a given candidate task set.We describe the candidate task set with a valid sequence and propose a sequence-based scheduling strategy.In order to improve the efficiency of finding a valid sequence,we design some pruning strategies and give the corresponding scheduling algorithm.Finally,we propose a timeout handling algorithm when some task runs timeout.Experiment shows that the total profit of the proposed algorithm is very close to the ideal maxima and is obviously bigger than related scheduling algorithms under different experimental settings.

关 键 词：MAPREDUCE 任务集 调度算法 利润 大数据 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]