基于大数据平台的网络日志机器行为分析  

Network log machine behavior analysis based on big data platform

作　　者：陈涵 张仰森[1,2] 何梓源 祁浩家 黄改娟 CHEN Han;ZHANG Yang-sen;HE Zi-yuan;QI Hao-jia;HUANG Gai-juan(Institute of Intelligent Information Processing,Beijing Information Science and Technology University,Beijing 100192,China;National Economic Security Early Warning Engineering Beijing Laboratory,Beijing Jiaotong University,Beijing 100044,China)

机构地区：[1]北京信息科技大学智能信息处理研究所,北京100192 [2]北京交通大学国家经济安全预警工程北京实验室,北京100044

出　　处：《计算机工程与设计》2023年第7期2232-2241,共10页Computer Engineering and Design

基　　金：国家自然科学基金项目(62176023)。

摘　　要：为从海量日志中分析并提取可能存在的机器行为日志,应用概率统计和K-means算法对网络日志进行分析,提出基于多阶段层次化判别的ChenRF模型,共包含两部分内容,分别为基于特征规则和阈值匹配的机器行为判定模型与基于Bi-LSTMAttention的机器行为识别模型。应用Spark批处理框架,对所提模型进行算法实现,构建以一小时为单位的分布式网络日志批处理分析平台。在1万和千万级别规模的数据上,对构建的大数据日志分析处理平台进行测试,验证了所提模型的有效性。To analyze and extract possible machine behavior logs from massive logs,probability statistics,K-means algorithm were applied to analyze network logs,and ChenRF model based on multi-stage hierarchical discrimination was proposed,which consisted of two parts,namely the machine behavior judgment model based on feature rules and threshold matching and the machine behavior recognition model based on BI-LSTMAttention.The Spark batch processing framework was used to implement the algorithm of the proposed model,and a distributed batch analysis platform for network logs was built in the unit of one hour.A test was carried out on the built big data log analysis and processing platform on the scale of 10 thousand and 10 million,which verifies the effectiveness of the proposed model.

关 键 词：网络日志 超文本传输协议 机器行为识别 数据分析与挖掘 深度学习 分布式存储技术 大数据计算技术 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]