基于MR的高可靠分布式数据流统计模型  

Statistical Model of Distrubuted Data Strem with High Reliability Based on MR

作　　者：朱蔚林 木伟民 金宗泽 王伟平[1] 

机构地区：[1]中国科学院信息工程研究所,北京100093 [2]中国科学院大学,北京100049

出　　处：《计算机技术与发展》2018年第1期6-10,16,共6页Computer Technology and Development

基　　金：国家自然科学基金(61402473)

摘　　要：结合流数据独有的特点,以数据流上基于窗口模型的连续分组统计为应用场景,结合现今主流的流数据处理平台Storm和Spark Streaming的优点,提出了一个高吞吐、低延迟、高可扩展性的分布式数据流统计模型Mars,解决由于流数据易失、时效性强造成的吞吐量压力大、数据延迟低等问题。在容错方面,Mars提供了at-least-once语义支持以防出现重大错误。采用真实实验环境对Mars进行测试,与目前流行的分布式流处理平台Spark Streaming和Storm相比,Mars对数据的实时性操作延迟介于二者之间,但就不同的集群规模而言,Mars的吞吐率明显优于二者1到2倍,就语义准确性而言,Mars实现了与Storm同级别的语义限制。According to the unique characteristics of the data stream, with consecutive grouping statistics based on window model in the data flow as application scenarios, combined with the advantages of mainstream stream data processing platform like Storm and Spark Streaming, we propose a distributed statistical model of data stream with high throughput and scalability as well as low latency,namely Mars. It solves the problems of strong throughput and low latency due to losing data easily and strong timelessness. On the fault-tolerant,Mars provides at -least-once semantic support against major errors. It is tested in real experiment environment and made a comparison with the currently pop- ular distributed flow processing platform Spark Streaming and Storm, which show that it is between them in real-time operation delay for data. However,in terms of the scale of the cluster,Mars＇ throughput rate is significantly better than that of the two,and in terms of semantic accuracy, it achieves the semantic limits of the same level as Storm.

关 键 词：数据流 分组统计 连续查询 分布式系统 实时处理 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]