基于MapReduce并行计算的网站日志数据分析处理  

Analysis and Processing of Website Log Data Based on MapReduce Parallel Computing

作　　者：刘义卿 陈新房 Liu Yiqing;Chen Xinfang(College of Information Engineering,Institute of Disaster Prevention,Sanhe 065201,China)

机构地区：[1]防灾科技学院信息工程学院,河北三河065201

出　　处：《科学技术创新》2022年第1期101-104,共4页Scientific and Technological Innovation

基　　金：2020年防灾科技学院教育研究与教学改革项目(JY2020A11);中央高校基本科研业务费项目(ZY20215126)。

摘　　要：随着信息技术的快速发展,大数据正以成倍的形势增长。数据也越来越呈现出数量巨大、结构复杂、类型众多、富有价值等特点。然而挖掘出数据中的价值才是关键,面对海量的数据,单靠传统的计算方式已经不能满足数据处理的需要。并行计算的优越性已经得到了的认可,MapReduce已经得到了广泛的应用。阐述MapReduce的原理基础,应用MapReduce计算模型的特性,将分布式文件系统中的大规模数据集切分成独立的数据块将计算均匀的分布到多台异构计算机上,来分析log日志,是一个很好的处理方案。大型网站的服务器往往会产生海量的log日志进行处理,这些log日志记录的其他机器访问服务器的IP、时间、HTTP协议、状态码等信息。With the rapid development of information technology,big data is growing exponentially.The data are increasingly showing the characteristics of huge quantity,complex structure,many types and rich value.However,mining the value of data is the key.In the face of massive data,traditional computing methods alone can not meet the needs of data processing.The superiority of parallel computing has been recognized,and MapReduce has been widely used.This paper expounds the principle basis of MapReduce,applies the characteristics of MapReduce calculation model,cuts the large-scale data set in the distributed file system into independent data blocks,and evenly distributes the calculation to multiple heterogeneous computers to analyze log logs.It is a good processing scheme.Servers of large websites often generate massive log logs for processing.These log logs record the IP,time,HTTP protocol,status code and other information of other machines accessing the server.

关 键 词：大数据 并行计算 MAPREDUCE 数据分析 日志 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]