支持大规模流数据处理的在线MapReduce数据传输机制  

Online MapReduce Data Transmission Mechanism Supporting Large-Scale Stream Data Processing

作　　者：魏晓辉[1] 李聪[1] 李洪亮[1] 李翔[1] 刘圆圆[1] 李丽娜[1] 庄园[1] 

机构地区：[1]吉林大学计算机科学与技术学院,长春130012

出　　处：《吉林大学学报（理学版）》2015年第2期273-279,共7页Journal of Jilin University:Science Edition

基　　金：国家自然科学基金(批准号:61170004);教育部高等学校博士学科点专项科研基金(批准号:20130061110052);吉林省科技发展计划重点科技攻关项目(批准号:20140204013GX);深部探测技术与实验研究专项基金(批准号:SinoProbe-09-01);吉林大学基本科研业务费专项基金(批准号:450060491439)

摘　　要：针对流数据规模参差不齐、流量动态变化且突发性较强的特点,提出一种可伸缩的动态MapReduce计算模型,支持大规模动/静态数据在线处理.基于Event推送方式,利用Netty底层异步通信方式等技术,建立在线MapReduce数据传输机制,进一步实现其原型程序,解决了大规模分布式计算程序的快速在线传输和数据分发等问题,支持流数据动态分发机制,为动态MapReduce模型提供支撑.与HadoopOnline系统的传统Socket管道传送方式相比,该方法能有效提高作业之间数据的传送效率,从而提高大规模流数据处理的实时性.We proposed a scalable and dynamic MapReduce computation model which supports the online processing of large-scale dynamic/static data against the characteristics of uneven stream data size and dynamic flowing and breaking out suddenly.On this basis,we proposed an online MapReduce data transmission mechanism and implemented its prototype program based on the push mode of Event and the use of Netty asynchronous communication technology.This paper focuses on solving fast online transfer of the large-scale distributed computing program and data dynamic distribution to provide support for dynamic MapReduce model.The experimental results show that the method can greatly improve the transmission efficiency of data between jobs compared with the traditional socket pipeline method in Hadoop system and improve real-time data stream handling significantly.

关 键 词：大数据 流数据处理 MAPREDUCE模型 数据传输机制 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]