面向MapReduce计算模式的中间数据通信优化  

Communication optimization for intermediate data of MapReduce computing model

作　　者：曹云鹏 王海峰[1,2] CAO Yunpeng;WANG Haifeng(College of Information Science and Engineering,Linyi University,Linyi Shandong 276002,China;Institute of Linyi University,Shandong Provincial Key Laboratory of Network Based Intelligent Computing,Linyi Shandong 276000,China)

机构地区：[1]临沂大学信息科学与工程学院,山东临沂276002 [2]山东省网络环境智能计算技术重点实验室临沂大学研究所单位问题已经过作者核实,山东临沂276002

出　　处：《计算机应用》2018年第4期1078-1083,共6页journal of Computer Applications

基　　金：山东省自然科学基金资助项目(ZR2017MF050,ZR2015FL014);山东省高等学校科学技术计划项目(J17KA049);英文没有，原文末中文也没有：山东省重大科技创新工程项目(2017CXGC0701);山东省自主创新及成果转化专项(2014ZZCX02702);山东省重点研发项目(2016GGX109001)。

摘　　要：针对MapReduce计算模式在Map阶段结束后会产生海量中间数据,导致存在大量跨越机架交换机的数据通信问题,提出一种优化Map密集型作业的中间数据通信优化方法。首先,提取MapReduce计算作业的运行前调度信息的特征并且量化数据通信活跃度;然后,采用朴素贝叶斯分类模型实现分类预测,将历史作业的运行数据作为样本来训练分类模型;最后,根据作业分类预测结果把通信活跃的作业集中映射到同一机架中,通过提高通信局部性来优化性能瓶颈。实验结果表明,所提方案对Shuffle子过程稠密的作业优化效果明显,能够提高4%~5%的计算性能;此外,在多用户运行情况下能降低4.1%中间数据通信延迟。所提方法可有效降低大数据计算过程中的通信延迟,提高异构集群的计算性能。Aiming at the communication problem of crossing the rack switches for a large amount of intermediate data generated after the Map phase in the MapReduce process,a new optimization method was proposed for the map-intensive jobs.Firstly,the features from the pre-running scheduling information were extracted and the data communication activity was quantified qualified.Then naive Bayesian classification model was used to realize the classification prediction by using the historical jobs running data to train the classification model.Finally,the jobs with active intermediate data communication process were mapped into the same rack to keep communication locality.The experimental results show that the proposed communication optimization scheme has a good effect on shuffle-intensive jobs,and the calculation performance can be improved by 4%-5%.In the case of multi-user multi-jobs environment,the intermediate data can be reduced by 4.1%.The proposed method can effectively reduce the communication latency in large-scale data processing and improve the performance of heterogeneous clusters.

关 键 词：MapReduce计算模型 大数据处理 通信优化 中间数据 机器学习 

分 类 号：TP302.7[自动化与计算机技术—计算机系统结构]