MapReduce Shuffle性能改进  

Performance improvement of MapReduce Shuffle

作　　者：熊倩[1] 张 郭明[1] 徐婕[1] 

机构地区：[1]湖北大学计算机与信息工程学院,武汉430062

出　　处：《计算机应用》2017年第A01期58-62,67,共6页journal of Computer Applications

基　　金：国家自然科学基金资助项目(61403132);湖北省自然科学基金资助项目(2015CFB404)

摘　　要：针对Reduce端从Map端拷贝大量结果数据耗时长的问题,提出一种新方案即对Map节点上同一个作业的多个Map任务所产生的大量临时结果数据进行合并,以取代原有Map Reduce架构对单个Map任务的结果数据作合并的机制。该方案减少了Map节点的输出结果数据量,能减少整个集群的网络传输数据量,节省Reduce端拷贝Map端输出数据的时间,从而减少了Map Reduce作业执行时间,提升了Map Reduce的执行性能。To solve the problem that Reduce end takes much time to copy result data from the Map end, in this paper, a new solution was proposed to merge temporary result data generated by multiple Map tasks on the Map node and replace that the original MapReduce integrates the results of the single Map task. The solution reduces the amount of output data of the Map node and data transmitted by the entire cluster, saves the output data copy time from Reduce end to the Map end, thereby decreasing the execution time of the entire MapReduce job and improving the execution performance of MapReduce.

关 键 词：HADOOP MAPREDUCE SHUFFLE 性能 

分 类 号：TP311.5[自动化与计算机技术—计算机软件与理论]