基于Bert和自适应聚类的在线日志解析方法  

Online Log Parsing Method Based on Bert and Adaptive Clustering

作　　者：卢家伟 卢士达 刘思思 吴承荣[1] LU Jiawei;LU Shida;LIU Sisi;WU Chengrong(School of Computer Science and Technology,Fudan University,Shanghai 200082,China;Engineering Research Centre of Network Information Security Audit and Monitoring of Ministry of Education,Fudan University,Shanghai 200082,China)

机构地区：[1]复旦大学计算机科学技术学院,上海200082 [2]复旦大学网络信息安全审计与监控教育部工程研究中心,上海200082

出　　处：《计算机科学》2024年第11期65-72,共8页Computer Science

基　　金：复旦大学网络信息安全审计与监控教育部工程研究中心与国家电网上海数据中心合作项目(09B307-9003001-0014-1)。

摘　　要：日志解析是一种从原始日志文件中提取有效信息的技术,它可以用于系统故障诊断、性能分析、安全审计等领域。日志解析的主要挑战在于日志数据的非结构化、多样性和动态性。不同的系统和应用程序可能使用不同的日志格式,随着时间的推移,日志格式也会发生变化。文中提出一种能够自适应不同日志源和日志格式变化的在线日志解析方法BertLP,它使用预训练语言模型Bert,并结合自适应聚类算法对日志中的单词进行静动态识别,从而对日志进行分组生成日志模板。BertLP方法不需要人工定义日志模板或正则表达式,也不需要对单词进行频率统计,而是通过学习日志消息的语义和结构特征,来自动识别日志字段和类型。在多个公开日志数据集上的对比实验显示,BertLP方法在日志解析的准确率上比现有最佳方法提高了6.1%,并且在日志解析任务上表现更好。Log parsing is a technique for extracting valid information from raw log files,which can be used in areas such as system troubleshooting,performance analysis and security auditing.The main challenge of log parsing is the unstructured,diversity and dynamics of log data.Different systems and applications may use different log formats,and log formats may change over time.Therefore,this paper proposes BertLP,an online log parsing method that can automatically adapt to different log sources and log format variations.It uses a pre-trained language model,Bert,combined with an adaptive clustering algorithm for static and dynamic recognition of words in logs to group logs to generate log templates.Instead of manually defining log templates or regular expressions and performing frequency counts on words,BertLP automatically identifies log fields and types by learning semantic and structural features of log message.Comparative experiments on public log datasets show BertLP improves log parsing accuracy by 6.1%compared with the best available method and performs better on log parsing tasks.

关 键 词：日志解析 Bert 自适应聚类 语义提取 

分 类 号：TP181[自动化与计算机技术—控制理论与控制工程]