基于Spark的网络日志分析平台研究与设计  

Research and implementation of network log analysis platform based on Spark

作　　者：林宗缪[1] 郭先超[1] 裴雨清 唐浩[1] 姚文勇[1] 

机构地区：[1]上海市质量监督检验技术研究院,上海201114

出　　处：《自动化与仪器仪表》2017年第11期157-159,162,共4页Automation & Instrumentation

摘　　要：从日志数据采集、存储、分析等层面研究分布式计算技术在网络日志分析中的应用。通过Flume采集海量网络日志,利用Spark并行计算及Spark Streaming流式计算技术搭建了网络日志分析平台。平台针对大数据在批量处理和实时计算时存在的问题提出优化的架构方式,解决了冲突问题。经实验表明,该平台能对大数据级别的网络日志进行实时与批量分析,而且具有高吞吐能力、高扩展性、高容错性的特点,符合设计目标。The paper studies applications of the distributed computing technology on network log analysis in the aspcct of data collection,storage and analysis. The paper design a network log analysis platform, on the basis of data collection technology of Flume, Spark parallel computing and stream computing technology of Spark Streaming. The platform presents an optimal archilecture to soh, e the pn）blem on batch anti real time hig data processing. Experimental result shows that,the platform can deal big data of network log by real time and batch analysis,and its key features of excellent performances, high scalability and high fault-tolerance satisfies the design re- quirement.

关 键 词：大数据 SPARK FLUME 日志分析 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]