基于文本策略和SMCS的海量日志分析方法  

Mass Log Analysis Method Based on File Strategy and SMCS

作　　者：张俊峰[1] 冯巧娟[1] 张晓丽[1,2] 

机构地区：[1]河南城建学院计算机科学与工程系,河南平顶山467036 [2]南京航空航天大学航空科技智能材料与结构重点实验室,南京210016

出　　处：《计算机工程》2012年第3期43-45,48,共4页Computer Engineering

基　　金：国家自然科学基金资助项目(60907038);河南省科技攻关计划基金资助重点项目(102102210020)

摘　　要：现有的海量日志统计分析方法速度慢,且对硬件配置的要求高。为此,提出一种基于文本策略和SMCS的海量日志分析方法。根据文件的软件设计策略,采用日志文件索引方法,将日志文件与日志时间关联,以加快日志提取。SMCS算法采用哈希表、文件归并、堆操作方法对海量日志进行统计分析和内存损耗控制。通过对真实软件进行对比实验,结果表明,该方法的分析速度比传统方法提高4倍。Now the statistical analysis for the massive logs not only is time-consuming, but also requires higher hard configuration. For these questions, a method based on the file software strategy and SMCS algorithm is proposed in this paper. By the software strategy, the log file indexing and the association between the log file and log time are adopted in this method, so as to quicken the log extraction. In order to analyze the massive logs statistically and control the memory loss, hash tables, file merging and heap operation are used in the SMCS algorithm. By experiment of the real software, results indicate that the proposed method is faster than that of traditional statistical analysis by four times.

关 键 词：Syslog日志 日志分析 SMCS算法 海量日志 文本策略 控制内存 

分 类 号：TP311.5[自动化与计算机技术—计算机软件与理论]