基于自主容器云平台的大数据日志采集系统  

Big Data Log Collection System Based on Self-controlled Container Platform

作　　者：吴鑫泉 杨军[1] WU Xin-quan;YANG Jun(Department 4 of Foundation, North China Institute of Computing Technology, Beijing 100083, China)

机构地区：[1]华北计算技术研究所基础四部,北京100083

出　　处：《计算机与现代化》2019年第2期102-106,共5页Computer and Modernization

摘　　要：随着云计算、虚拟化、容器云等技术的应用以及国产自主可控的要求,越来越多的服务会部署在国产的服务器以及自主可控的容器云平台上。自主可控云平台上的服务在运行的过程中,需要获取日志数据,来进行一系列的监控、统计、分析以及预测工作。然而由于国产服务器的特殊性以及容器云平台的特殊性,传统的日志采集方式(包含实时采集和集中式采集)并不能够很好地适用于自主可控的容器云平台,为了提高服务的质量和性能并且保证日志数据的一定全面性,本文提出一种自适应的数据采集算法,能够根据服务器的负载自我调整日志数据采集发送量。当服务器负载较低的时候,提高数据采集和发送的数量,甚至是做到实时采集;当服务器负载较高时,降低对日志数据的采集和发送量,从而降低对服务器负载的压力,提高服务自身的质量和性能。最后通过理论和实验分析验证该数据采集算法在保证数据一定的全面性的同时,能有效缓解国产服务器的压力。With the application of technologies such as cloud computing, virtualization, container cloud, and domestically-controlled and controllable requirements, more and more services will be deployed on domestic servers and self-controlled container cloud. In the process of running the service, we need to obtain log data for a series of monitoring, statistics, analysis and forecasting work. However, due to the particularity of the domestic server and the particularity of self-controlled container cloud, the traditional log collection methods (including real-time acquisition and centralized acquisition) are not well suited for self-controlled container cloud. In order to improve the quality and performance of the service and ensure a certain comprehensiveness of the log data, an adaptive data acquisition algorithm is proposed, which can self-adjust the log data collection and transmission according to the load of the server. When the server load is low, the number of data collection and transmission is increased, even in real-time collection;when the server load is high, the collection and transmission of log data is reduced, thereby reducing the pressure on the server load, and improving the quality and performance of the service itself. Finally, theoretical and experimental analysis proves that the data acquisition algorithm can effectively alleviate the pressure of domestic servers while ensuring a certain comprehensiveness of data.

关 键 词：自主可控 容器云 数据采集 大数据 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]