基于Hadoop的网络日志挖掘方案的设计  

Design of Web log mining scheme based on Hadoop

作　　者：许抗震 吴云[1] XU Kangzhen;WU Yun(College of Computer Science and Technology,Guizhou University,Guiyang 550025,China)

机构地区：[1]贵州大学计算机科学与技术学院,贵州贵阳550025

出　　处：《现代电子技术》2017年第9期115-120,共6页Modern Electronics Technique

基　　金：国家自然科学基金项目(NSF61370161);贵州省科学技术基金项目(黔科合J字[2010]2100);贵州大学博士基金项目(贵大人基合字(2009)029)

摘　　要：提出一种挖掘指数级别网络日志数据的解决思路,设计了一个高可靠的网络日志数据挖掘方案。针对现有的公开网络日志数据集,在数据预处理阶段实现了基于Map Reduce的过滤算法,并且挖掘出支持企业决策的服务信息。对该方案搭建的平台进行优化操作,性能提升了3.26%,最后对方案的高可靠性、日志文件个数对平台I/O速度的影响、平台和单机在查询性能上的对比等方面做了实验。结果表明:该设计方案不仅可靠,而且随着日志文件个数的翻倍增加,读操作耗时平均增加52.58%,写操作耗时平均增加79.69%。随着日志量的增加,单机的查询耗时急剧增长,而平台的查询耗时趋于稳定。随着机器节点的增加,运算耗时以平均8.87%的速度减少。A thought of mining the Web log data with exponent level is put forward.A high reliability Web log data mining scheme was designed.Aiming at the available public Web log dataset,the filtering algorithm based on MapReduce was implemented in the data preprocessing stage to mine the service information supporting the enterprise decision.The platform established with this scheme is optimized,and its performance is increased by3.26%.The effect of the scheme′s high reliability and log file quantity on the I/O speed of the platform,and the comparison of the platform with the single machine in the aspect of query performance were tested.The results show that the designed scheme is reliable,double increased with the increase of the log file quantity,the time cost of the read operation is increased by52.58%averagely,and the time cost of the write operation is increased by79.69%.With the increase of the log quantity,the query time cost of the single machine is increased rapidly,and the query time cost of the platform is stable.With the increase of the machine nodes,the computational time cost is decreased by8.87%averagely.

关 键 词：网络日志 数据挖掘 数据清洗 HADOOP MYSQL 

分 类 号：TN711-34[电子电信—电路与系统] TP391.9[自动化与计算机技术—计算机应用技术]