基于Spark Streaming的实时数据采集分析系统设计  

Design of Real-time Data Collection and Analysis System Based on Spark Streaming

作　　者：党寿江[1,2] 刘学[1] 王星凯[1,2] 刘春梅[1] 

机构地区：[1]中国科学院声学研究所国家网络新媒体工程技术研究中心,北京100190 [2]中国科学院大学,北京100049

出　　处：《网络新媒体技术》2017年第5期48-53,共6页Network New Media Technology

基　　金：中国科学院战略性先导科技专项:新型传播技术研究与系统研制(XDA06040602)

摘　　要：大数据量的实时数据分析系统,需要快速的处理和响应。为了实现实时数据分析,本文设计了基于Spark Streaming的实时数据采集分析系统,并对有状态计算操作的基数计算的精确计算方法和估算方法进行了比较分析。实验表明,基于HyperLogLog++的基数估算方法在处理时间和存储占用空间上有明显优势,而计算偏差基本可以忽略不计,更适于大数据的基数估算。Real - time data analysis systems with large amounts of data require fast processing and response. In order to realize real - time data analysis, this article designs a real - time data collection and analysis system based on Spark Streaming and analyzes the per- formance between exactly and approximately counting in cardinality counting for the stateful operation. Experiments show that the cardi- nality estimation method based on HyperLogLog ＋ ＋ has obvious advantages in processing time and storage space, and the impact caused by relative deviation is negligible, which is more suitable for the cardinality counting in real - time large data analysis.

关 键 词：基数计算 实时数据分析 Spark流式处理 不重复计数 HyperLogLog++ 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]