面向大规模流数据的可扩展分布式实时处理方法  

A Scalable and Distributed Method for Processing Large-Scale Data Streams in Real-Time

作　　者：蔡斌雷[1] 郭芹[2] 朱世伟[1] 任家东[3] CAI Binlei GUO Qin ZHU Shiwei REN Jiadong(Information Research Institute, Shandong Academy of Sciences, J inan 250014, China Quancheng College, University of Jinan, Yantai 265600, China College of Information Science and Engineering, Yanshan University, Qinhuangdao 066004,China)

机构地区：[1]山东省科学院情报研究所,山东济南250014 [2]济南大学泉城学院,山东烟台265600 [3]燕山大学信息科学与工程学院,河北秦皇岛066004

出　　处：《青岛科技大学学报（自然科学版）》2016年第5期584-590,共7页Journal of Qingdao University of Science and Technology:Natural Science Edition

基　　金：国家自然科学基金资助项目(61170190);山东省科技发展计划项目(2014GGX101013;2015GGX101032)

摘　　要：MapReduce是处理大规模数据集的常用技术,但不能满足大规模数据集中流数据实时计算的要求。对此提出一种面向大规模流数据的可扩展、分布式实时处理方法。该方法在Map阶段,建立基于内存Hash B+树的缓存结构对中间结果处理机制进行优化,以降低对中间结果的频繁读写造成的I/O消耗,同时消除对中间结果的排序,以降低对CPU的消耗;在Reduce阶段,设计基于动态增量Hash技术的快速内存处理方法,并消除对中间结果的多遍扫描合并,对流数据进行增量处理、单遍分析,以提高对流数据的实时分析能力。实验结果表明:上述方法可以对大规模流数据进行实时性处理,并且具有较好的可扩展性。MapReduce is a widely used technique for processing massive datasets,however,it is unable to support the real-time processing for large scale data streams.In this paper,we studied a scalable and distributed method,called SDRT-MR,based on MapReduce model,to process large scale data streams in real time.To lower the I/O cost and efficiently utilize CPU,a memory caching mechanism using Hash B＋tree is adopted to optimize the processing mechanism of intermediate results.To boost incremental one-pass analytics of data streams processing,we develop dynamic incremental hash techniques to support fast in-memory processing,simultaneously employ an efficient technique to identify frequent keys.Our experimental results on synthetic datasets show that SDRT-MR has higher real-time performance and better scalability.

关 键 词：大数据 分布式计算 流数据处理 MAPREDUCE 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]