面向云数据中心多语法日志通用异常检测机制  

Unified Anomaly Detection for Syntactically Diverse Logs in Cloud Datacenter

作　　者：张圣林 李东闻 孙永谦 孟伟彬 张宇哲 张玉志 刘莹[3,4] 裴丹 Zhang Shenglin;Li Dongwen;Sun Yongqian;Meng Weibin;Zhang Yuzhe;Zhang Yuzhi;Liu Ying;Pei Dan(College of Software,Nankai University,Tianjin 300350;Department of Computer Science and Technology,Tsinghua University,Beijing 100084;Institute for Network Sciences and Cyberspace,Tsinghua University,Beijing 100084;Beijing National Research Center for Information Science and Technology,Beijing 100084)

机构地区：[1]南开大学软件学院,天津300350 [2]清华大学计算机科学与技术系,北京100084 [3]清华大学网络科学与网络空间研究院,北京100084 [4]北京信息科学与技术国家研究中心,北京100084

出　　处：《计算机研究与发展》2020年第4期778-790,共13页Journal of Computer Research and Development

基　　金：国家重点研发计划项目(2018YFB0204304)。

摘　　要：得益于自然语言处理和机器学习方法的快速发展,基于日志对云数据中心软硬件系统进行自动异常检测变得越来越流行.无监督学习方法不需要标记异常日志,但通常存在准确性较低、仍需标注大量正常日志的问题.尽管有监督学习方法的准确性较高,但由于不同软硬件系统产生不同类型的、语法各异的日志,导致有监督学习方法需要为每一类型日志标注足够多的异常日志以训练相应的异常检测模型,这极大地增加了标注异常日志的人力成本.与此同时,不同类型日志在发生异常时往往具有相同或相似的语义.因此,提出了一种跨日志类型的通用异常检测机制——LogMerge.该机制通过学习多语法日志的语义相似性,可实现日志异常模式的跨日志类型迁移,从而大大减少了异常标注开销.LogMerge采用词嵌入方法先后构建单词和模板的向量,然后使用聚类方法将语义相同或相近的模板聚成一类,解决了不同类型日志语法不同带来的挑战.此外,LogMerge结合CNN与LSTM方法构建异常检测模型,既有效提取了日志序列的前后依赖性,又显著降低了日志序列中噪声带来的影响.使用公开日志数据集的实验表明,相比于当前的有监督学习方法和无监督学习方法,LogMerge取得了更高的准确性.实验还验证了LogMerge能够显著减少异常标注工作量——在目标类型日志异常标注较少时,依然能够取得较高的准确性.Benefit from the rapid development of natural language processing and machine learning methods,log based automatic anomaly detection is becoming increasingly popular for the software and hardware systems in cloud datacenters.Current unsupervised learning methods,requiring no labelled anomalies,still need to obtain a large number of normal logs and generally suffer from low accuracy.Although current supervised learning methods are accurate,they need much labelling efforts.This is because the syntax of different types of logs generated by different software hardware systems varies greatly,and thus for each type of logs,supervised methods need sufficient anomaly labels to train its corresponding anomaly detection model.Meanwhile,different types of logs usually have the same or similar semantics when anomalies occur.In this paper,we propose LogMerge,which learns the semantic similarity among different types of logs and then transfers anomaly patterns across these logs.In this way,labelling efforts are reduced significantly.LogMerge employs a word embedding method to construct the vectors of words and templates,and then utilizes a clustering technique to group templates based on semantics,addressing the challenge that different types of logs are different in syntax.In addition,LogMerge combines CNN and LSTM to build an anomaly detection model,which not only effectively extracts the sequential feature of logs,but also minimizes the impact of noises in logs.We have conducted extensive experiments on publicly available datasets,which demonstrates that compared with the current supervised unsupervised learning methods,LogMerge achieves higher accuracy.Moreover,LogMerge achieves high accuracy when there are few anomaly labels in the target type of logs,which therefore significantly reduces labelling efforts.

关 键 词：日志 异常检测 云数据中心 词嵌入 CNN LSTM 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]