MapReduce中shuffle优化与重构  

Optimization and reconstruction shuffle in MapReduce

作　　者：彭辅权[1] 金苍宏[1] 吴明晖[2] 应晶[1,2] 

机构地区：[1]浙江大学计算机学院,杭州310027 [2]浙江大学城市学院,杭州310015

出　　处：《中国科技论文》2012年第4期241-245,共5页China Sciencepaper

基　　金：清华-腾讯互联网创新技术联合实验室资助项目(2011-8)

摘　　要：详细介绍了MapReduce编程框架,具体分析了MapReduce中shuffle阶段流程。分别从Map端数据压缩、重构远程数据拷贝传输协议、Reduce端内存分配优化三方面来优化和重构Shuffle。最后通过搭建Hadoop集群,运用MapReduce分布式算法测试实验数据。实验结果证明优化重构后的shuffle能显著提高MapReduce计算性能。We describe the MapReduce programming framework in detail,and analyze the shuffle-stage process.Shuffle in MapReduce is optimized and reconstructed through the following three measures：compressing the output of the Map end,reconstructing the protocol used to copy the data form the Map end to the Reduce end,and optimizing memory allocation on the Reduce end.Finally,through building a Hadoop cluster,the experimental data are tested using the MapReduce distributed algorithm.Experimental results show that the MapReduce computing performance improves significantly after optimizing the reconstructed shuffle.

关 键 词：云计算 HADOOP MAPREDUCE SHUFFLE 

分 类 号：TP311.1[自动化与计算机技术—计算机软件与理论]