MapReduce中连接负载均衡优化研究  

Optimizing load balancing of joins in MapReduce

作　　者：翟红敏[1,2] 刘国华[1,2] 赵威[1,2] 刘源源[1,2] 翟红坤 

机构地区：[1]东华大学计算机科学与技术学院,上海201620 [2]国网黑龙江省电力有限公司信息通信公司,黑龙江哈尔滨150000

出　　处：《计算机工程与科学》2014年第10期1860-1865,共6页Computer Engineering & Science

基　　金：国家自然科学基金资助项目(61070032)

摘　　要：数据分析和处理是大规模分布式数据处理应用中的重要任务。由于简单易用和具有灵活性,MapReduce编程模型逐渐成为大规模分布式数据处理系统(如Hadoop系统)的核心模型。由于所处理的数据可能不是均匀分布的,MapReduce编程模型在处理连接操作时,会出现数据倾斜问题。数据倾斜问题严重降低了MapReduce执行连接操作的效率。针对MapReduce中连接操作的数据倾斜问题,分析了造成MapReduce连接性能瓶颈的原因并建立负载均衡代价模型,提出了用范围分割方法控制连接过程中的数据倾斜问题实现负载均衡的策略。实验结果表明,所提方法明显提高了连接的效率。Data analysis and processing is one of the most important tasks in large-scale distributed data processing applications. Due to its simplicity and scalability, MapReduce programming model has gradually become the crucial model for large-scale distributed data processing systems （eg. Hadoop）. Since the data may be uniformly distributed, data skew occurs when MapReduce programming model joins data,thus degrading the join performance severely. To solve data skew, its reason is analyzed, the load balancing cost model is established, and the rangepartitioner algorithm is proposed to control data skew so as to realize load balancing. Experimental results demonstrate that our method can obviously im- prove the efficiency of joins.

关 键 词：MAPREDUCE 连接 数据倾斜 范围分割 负载均衡 

分 类 号：TP391.9[自动化与计算机技术—计算机应用技术]