MapReduce中Combine优化机制的利用  

Application of Combine optimize mechanisms in MapReduce

作　　者：贾欧阳[1] 阮树骅[1] 田兴[1] 杨峻兴[1] 李丹[1] 

机构地区：[1]四川大学计算机学院,四川成都610065

出　　处：《计算机时代》2013年第9期1-4,共4页Computer Era

摘　　要：由Apache软件基金会开发的Hadoop分布式系统基础架构,作为一个主流的云计算平台,其核心框架之一的MapReduce性能已经成为一个研究热点,其中对于Shuffle阶段的优化,使用Combine优化机制是关键。文章详细介绍了MapReduce计算框架及Shuffle流程;分别从机理简介、执行时机、运行条件三方面详细阐述了如何利用Combine优化机制;通过搭建Hadoop集群,运用MapReduce分布式算法测试实验数据。实验结果充分证明,正确地运用Combine优化机制能显著提高MapReduce框架的性能。The Hadoop, a distributed system infrastructure developed by the Apache Software Foundation, has become a mainstream on cloud-computing platform. How to improve its performance has become a hot issue, of which key points are the optimization of Shuffle stage, and the Combine optimization mechanism. The MapReduce computation framework and shuffle process are carefully introduced, as well as how to use the Combine optimization mechanism by mechanism profiling, execution timing and running condition, finally tests experimental data by using MapReduce distributed algorithms. The cluster results demonstrate that to use Combine optimized mechanism correctly can significantly improve the performance of the MapReduce framework.

关 键 词：云计算 HADOOP MAPREDUCE SHUFFLE COMBINE 

分 类 号：TP393.2[自动化与计算机技术—计算机应用技术]