Hadoop平台下加权马氏距离的Web大数据分析研究  

A Web Big Data Analysis Research Based on Hadoop Platform and Weighted Mahalanobis Distance

作　　者：王艳洁 范存群[2] WANG Yanjie;FAN Cunqun(Quzhou College of Technology,Quzhou 324000,China;National Satellite Meteorological Center,Beijing 100081,China)

机构地区：[1]衢州职业技术学院,浙江衢州324000 [2]国家卫星气象中心,北京100081

出　　处：《电视技术》2018年第11期67-71,共5页Video Engineering

基　　金：衢州职业技术学院校科研项目(QZYY1721);衢州市科技计划指导性项目(2018004)

摘　　要：Web大数据具有数据量大、数据异构性强、挖掘难度大等问题,针对如何提高Web大数据聚类分析准确性并保证数据分析的时效性,本文提出一种基于Hadoop平台和加权马氏距离的Web大数据分析方法。该方法在基于Hadoop平台架构上首先对Web大数据进行预处理和数据切片,然后通过计算得到方差贡献率并作为马氏距离计算的权重,最后通过加权马氏距离计算对数据判定聚类。仿真实验表明,所提方法可以有效地保证Web大数据聚类分析的准确性并具有较高的时效性。Web big data has problems such as large amount of data,strong data heterogeneity,and difficulty in mining.In order to improve the accuracy of Web big data clustering analysis and guarantee the timeliness of data analysis,this paper proposed a Web big data analysis method based on Hadoop platform and weighted Mahalanobis Distance.Based on Hadoop platform architecture,this method first preprocessed and slicing Web big data,then calculated the variance contribution rate and calculates the weight of Mahalanobis distance.Finally,the data was clustered by weighted Mahalanobis distance calculation.Simulation experiments showed that the proposed method can effectively guarantee the accuracy of Web big data clustering analysis and has high timeliness.

关 键 词：HADOOP平台 Web大数据 加权马氏距离 数据处理 方差贡献率 判定聚类 

分 类 号：TP391.9[自动化与计算机技术—计算机应用技术]