基于Spark Streaming的实时数据分析系统及其应用  

Real-time data analysis system based on Spark Streaming and its application

作　　者：韩德志[1] 陈旭光[1] 雷雨馨 戴永涛[1] 张肖[1] HAN Dezhi CHEN Xuguang LEI Yuxin DAI Yongtaol ZHANG Xiao(College of Information Engineering, Shanghai Maritime University, Shanghai 201306, China School of Information Engineering, Zhengzhou University, Zhengzhou Henan 450001, China)

机构地区：[1]上海海事大学信息工程学院,上海201306 [2]郑州大学信息工程学院,郑州450001

出　　处：《计算机应用》2017年第5期1263-1269,共7页journal of Computer Applications

基　　金：国家自然科学基金资助项目(61373028;61672338)~~

摘　　要：为了实现对实时网络数据流的快速分析,设计一种分布式实时数据流分析系统(DRDAS),能有效解决并发访问数据流的收集、存储和实时分析问题,为大数据环境的网络安全检测提供了一种有效的数据分析平台;根据Spark Streaming运行的原理设计一种动态采样的K-Means并行算法,与DRDAS结合能实时有效地检测大数据环境下的各种分布式拒绝服务(DDo S)攻击。实验结果显示:DRDAS具有好的可扩展性、容错性和实时处理能力,与动态采样的K-Means并行算法结合能实时地检测各种DDo S攻击,缩短了攻击的检测时间。In order to realize the rapid analysis of massive real-time data, a Distributed Real-time Data Analysis System （DRDAS） was designed, which resolved the collection, storage and real-time analysis for mass concurrent data. And according to the operation principle of Spark Streaming, a dynamic sampling K-means parallel algorithm was proposed, which could quickly and efficiently detect all kinds of DDoS （ Distributed Denial of Service） attacks. The experimental results show that the DRDAS has good scalability, fault tolerance and real-time processing ability, and along with new K-means parallel algorithm, the DRDAS can real-time detect various DDoS attacks, and shorten the detecting time of attacks.

关 键 词：SPARK Streaming框架 分布式流处理 网络数据分析 分布式拒绝服务攻击 

分 类 号：TP316.2[自动化与计算机技术—计算机软件与理论]