分布式系统中的日志分析及应用  

Log analysis for distributed systems and its application

作　　者：陆杰 李丰 李炼[1,2] Lu Jie;Li Feng;Li Lian(State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190;Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100193;University of Chinese Academy of Sciences, Beijing 100190)

机构地区：[1]中国科学院计算技术研究所计算机体系国家重点实验室,北京100190 [2]中国科学院大学,北京100190 [3]中国科学院信息工程研究所,北京100193

出　　处：《高技术通讯》2019年第4期303-320,共18页Chinese High Technology Letters

基　　金：国家重点研发计划(2017YFB0202002);国家自然科学基金(61521092)资助项目

摘　　要：分布式系统是支撑当前大数据时代各种大数据应用和在线服务的基础平台,分布式系统的质量是大数据应用提供良好服务的基础和前提。伴随着大规模分布式系统的广泛应用,由分布式系统缺陷带来的影响和危害日益严重。但分布式系统在设计、实现和部署方面的复杂性,导致系统的开发和维护人员很难准确地理解和掌握整个系统的行为,难以及时发现系统中存在的故障并进行修复。分布式系统日志涵盖了丰富的信息,是辅助用户理解分布式系统逻辑、剖析系统性能、检测系统异常以及诊断故障原因的重要依据。但复杂的日志结构、庞大的日志规模以及属于不同功能模块、不同用户请求的日志之间相互交错,为人工分析、挖掘日志中的有效信息带来了巨大的困难。本文对近年来针对分布式系统日志的分析和应用技术进行综述:首先总结了分布式系统日志分析与应用的通用流程,提炼出其中的3个关键步骤,即日志的收集与解析、日志划分、以及日志特征的挖掘与应用;然后针对上述3个关键步骤,逐一分析需要解决的技术问题,分类阐述目前主流的技术方案,对比技术特征或适用场景。文章还归纳了目前常用的3类日志特征,并从4个方面就该领域未来可能的研究方向提出展望。Distributed systems are the key infrastructure in the big data era, it is vital to ensure their reliability since faults in the infrastructure can often lead to catastrophic failures. However, it is also very challenging, due to the complexity in its design, implementation and deployment. Log analysis is an effective way to help address the problem. Distributed systems often output logs with various runtime information. Such information can help users to understand system behaviour, profile system performance, detect anomaly and diagnose faults. However, log messages are unstructured and the unstructured logs from different user requests and different running threads are often intertwined with each other, all located in a large-sized log file. Therefore, it is very difficult for users to manually inspect the large log files and mine useful information in these files. Log analysis automatically processes the vast log data, providing user-friendly and useful information to the users, to help them understand system behavior, monitor system execution, and detect anomalies. This paper surveys various log analysis techniques. We summarize the basic process of log analysis into three main steps: log parsing and collection, log partition and log mining. We elaborate the key problems in each step and summarise their corresponding solutions. We describe three kinds of different features in logs which can be mined for different use cases. At last, we highlight the future research directions in this field.

关 键 词：分布式系统 日志分析 特征挖掘 异常检测 故障诊断 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]