一种基于分布式的启发式系统日志解析算法  

A distributed heuristic system log parsing algorithm

作　　者：万洪杰 马骏臣 孙国梓[1,2] WAN Hongjie;MA Junchen;SUN Guozi(School of Computer Science,Nanjing University of Posts and Telecommunications,Nanjing 210023,China;Institute of Applied Technology for Data Security Compliance,Nanjing University of Posts and Telecommunications,Nanjing 210023,China)

机构地区：[1]南京邮电大学计算机学院,江苏南京210023 [2]南京邮电大学数据安全合规应用技术研究所,江苏南京210023

出　　处：《南京邮电大学学报（自然科学版）》2023年第6期91-101,共11页Journal of Nanjing University of Posts and Telecommunications：Natural Science Edition

基　　金：国家自然科学基金(61906099);国家自然科学基金青年基金(62102192);国家博士后基金(2022M710071)资助项目。

摘　　要：日志在系统管理中被广泛应用于可靠性保证,是检测系统是否异常的重要数据源。在系统日志异常检测流程之前需要将原始的文本结构日志解析成异常检测模型能够识别的结构化日志,为了完成日志解析这一前置任务,提出了很多根据日志特性设计的日志解析算法,从而达到提取结构化日志的目的。针对已有的日志解析算法的不足之处作出改进,以解析效率作为突破点提出了基于分布式的系统日志解析任务分配模型和基于启发式的系统日志解析算法,通过组合分配模型和解析算法完成日志解析任务。利用HDFS和BGL等5种数据集实验验证了模型和算法的可行性和有效性,缩短了日志解析所需花费的时间,提高了日志解析的效率。Logs are widely used in system management and crucial to abnormal detection and system reliability.Before the abnormal detection on system logs,it is necessary to parse the original texts into structured logs that can be understood by the abnormal detection model.Many log parsing algorithms have been designed to extract structured logs,but certain shortcomings still exist.In this regard,this paper proposes a system log parsing model.This model constructs a log parsing task assignment method for distributed systems,and then establishes a heuristic log parsing algorithm.Thus,the log parsing task can be completed by combining the distributed assignment method and the heuristic parsing algorithm.Five kinds of data sets,such as HDFS and BGL,are used to verify the feasibility and effectiveness of the model,and the results demonstrate that the proposed model can reduce the time and improves the efficiency for log parsing.

关 键 词：日志 分布式系统 解析算法 任务分配 启发式 

分 类 号：TP303[自动化与计算机技术—计算机系统结构]