面向MapReduce计算的大规模集群通信优化  

Communication optimization for large-scale cluster aiming MapReduce computing

作　　者：曹云鹏 王海峰[1,2] 刘海涛 何淑庆[1,2] Cao Yunpeng;Wang Haifeng;Liu Haitao;He Shuqing(School of Information Science&Engineering,Linyi University,Linyi Shandong 276000,China;Shandong Provincial Key Laboratory of Network-based Intelligent Computing,Linda Institute,Linyi Shandong 276000,China)

机构地区：[1]临沂大学信息科学与工程学院,山东临沂276000 [2]临沂大学研究所山东省网络环境智能计算技术重点实验室,山东临沂276000

出　　处：《计算机应用研究》2020年第4期1174-1178,共5页Application Research of Computers

基　　金：山东省自然科学基金面上项目(ZR2017MF050);山东省高等学校科学技术计划项目(J17KA049);山东省重点研发项目(2019GGX1005,2018GGX101005,2017CXGC0701,2016GGX109001)。

摘　　要：为了优化大规模集群运行MapReduce作业时的通信效率和减少shuffle数据传输量,首先采用存储局部性换取通信局部性的策略建立一个分布式协同数据映射模型;其次通过随机抽样和机器学习方法来提取作业数据的局部性特征,实现map计算数据的有效部署;最后,利用软件定义网络的全局灵活控制能力,优选通信链路好的节点并将计算任务映射到该类节点中。实验表明对于中间数据混洗密集类作业有较好的优化效果,通信延迟降低了4.3%~5.8%。该方案能减少shuffle流量和数据迁移延迟,并且适合各种调度策略和网络拓扑结构。To optimize communication efficiency and reduce the data transmission of shuffle in large-scale clusters running for MapReduce jobs,this paper built a distributed collaborative data mapping model by replacing the communication locality using storage locality.Then it extracted the local features of jobs by random sampling and machine learning method in order to realize the effective deployment of map tasks.Finally,it selected the nodes with good communication links based on the software define network technology due to its global flexible control capabilities,and scheduled the map tasks to such nodes.Experimental results show that the model has better optimization effect on shuffle-intensive jobs.The communication delay is reduced by 4.3% to 5.8%.This solution can reduce shuffle traffic and data migration delay and it is suitable for various scheduling strategies and network topologies.

关 键 词：数据通信优化 MAPREDUCE 软件定义网络 协同数据映射 

分 类 号：TN915.02[电子电信—通信与信息系统]