基于Hadoop的在线网络日志分析系统研究  

RESEARCH ON HADOOP-BASED ONLINE NETWORK LOG ANALYSIS SYSTEM

作　　者：杨锋英[1] 刘会超[2] 

机构地区：[1]黄淮学院信息工程学院,河南驻马店463000 [2]黄淮学院网络信息管理中心,河南驻马店463000

出　　处：《计算机应用与软件》2014年第8期311-316,共6页Computer Applications and Software

基　　金：河南省科技攻关计划项目(122102310474)

摘　　要：网络日志分析对网络安全管理具有重要意义,但现有的网络日志分析系统具有无法处理海量日志数据、采用离线模式、处理时延较长等弊端。因此,提出网络日志数据的分级归档机制,实现海量数据的分级优化存储功能。通过将传统的网络日志分析模式和基于Hadoop的离线处理模式相结合,设计基于Hadoop的在线网络日志分析系统,并深入分析系统的功能架构和业务处理流程。实验结果表明:该系统具有良好的可扩展性,可以有效存储处理海量日志数据,具有较快的请求响应速度,并能够满足在线处理的要求。从而验证了系统设计的可行性和有效性。Network log analysis has vital significance to network security management,however existing network log analysis system has some drawbacks,such as unable to deal with huge amounts of log data,using offline mode,and longer processing delay,etc. Therefore,we propose the hierarchical archiving mechanism of network log data and realise the function of hierarchical and optimised storage of large amount of log data. Through the combination of traditional log analysis mode and Hadoop-based offline analysis mode,we design the Hadoop-based online log analysis system. Moreover,we analyse in depth the functional architecture and the transactions processing flow of the system.Experimental results show that the new system has good scalability and can effectively store and process massive log data. Furthermore,it has faster request and response speed which can meet the requirements of online processing. The feasibility and effectiveness of the system design is also verified.

关 键 词：网络安全 日志分析 HADOOP 文件分级归档 在线系统 

分 类 号：TP311.1[自动化与计算机技术—计算机软件与理论]