数据倾斜情况下基于MapReduce模型的连接算法研究  

Research on Data Skew Join Algorithm Based on MapReduce Model

作　　者：金健[1] 陈群[1] 赵保学[1] 

机构地区：[1]西北工业大学计算机学院,陕西西安710072

出　　处：《计算机与现代化》2013年第5期22-27,共6页Computer and Modernization

基　　金：国家"九七三"重点基础研究发展规划基金资助项目(2012CB316203);国家自然科学基金重点资助项目(61033007);国家"八六三"高技术研究发展基金资助项目(2012AA011004)

摘　　要：基于MapReduce的连接算法的研究是海量数据研究领域的一个重要内容,但都集中在数据分布均匀的情况下进行算法优化,而在实际应用中数据分布往往是不均匀的。本文基于此背景,提出一种适合在数据严重倾斜时使用基于MapReduce编程模型的连接算法Skew Control Join,算法通过采样获取数据集的整体分布,通过全局分区将数据集进行分割,使倾斜数据的处理平均分配到所有的Reduce任务上。实验表明在数据倾斜时,本文提出的算法具有良好的性能,达到研究目标。The study of join algorithm based on MapReduce is a hot topic in massive data research area. However, most current optimization work is based on the assumption that the data are evenly distributed. In practical applications, the data to be processed are often skew in distribution. This paper proposes a MapReduce join algorithm called Skew Control Join, which is adaptive for serious skew data. The algorithm gets the overall data distribution by sampling, the partitions the data by total partitioner to distribute the data evenly to all Reduce tasks. Experiment results show that the algorithm is of good performance when the pro- cessed data are skew.

关 键 词：连接算法 数据倾斜 全局分区 采样 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]