基于Hadoop平台的日志分析模型  

Hadoop platform based log analysis model

作　　者：于兆良[1] 张文涛[1] 葛慧[1] 艾伟[1] 孙运乾[1] 

机构地区：[1]中国航天工程咨询中心,北京100048

出　　处：《计算机工程与设计》2016年第2期338-344,428,共8页Computer Engineering and Design

摘　　要：为提高企业网络内海量日志数据的分析效率,构建基于Hadoop平台的日志分析模型。对模型框架进行总体设计,提出一种MapReduce编程模式的Apriori并行化算法,基于该算法对历史日志进行数据挖掘分析,计算用户行为的频繁模式,建立用户正常行为规则库,将实时日志与规则库中的规则进行模式匹配,实现对用户异常行为的检测。实验结果表明,该模型算法明显提高了日志分析效率。To improve the efficiency of analyzing mass log data in enterprise＇s network,a log analysis model was constructed based on Hadoop platform.The framework of the model was designed.A parallelization algorithm of Apriori was proposed in MapReduce programming mode.The historical log data were analyzed by data mining based on this algorithm and the frequent patterns of users＇ action were figured out to establish the rule base of users＇ normal actions.The real-time log was detected to find out anomalous actions of users through matching the rules of rule base.Experimental results show that the proposed algorithm of the model can improve the efficiency of log analysis significantly.

关 键 词：HADOOP平台 日志分析 MapReduce编程模式 APRIORI算法 数据挖掘 并行化 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]