基于布隆过滤算法的五台山数据分析应用研究  

Application Research of Wutaishan Data Analysis Based on Boolm Filter Algorithm

作　　者：赵青杉[1] 段宗娟 胡玉兰[1] ZHAO Qingshan;DUAN Zongjuan;HU Yulan(Department of Computer,Xinzhou Teachers University,Xinzhou 034000,China)

机构地区：[1]忻州师范学院计算机系,山西忻州034000

出　　处：《忻州师范学院学报》2020年第5期1-4,共4页Journal of Xinzhou Teachers University

基　　金：山西省大学生科技创新项目(2019511)。

摘　　要：随着互联网的发展,网络信息呈爆炸性增长,其中有很多有用的信息,但是也充斥着许多重复的信息。针对这一问题,提出了一种基于布隆过滤器算法的数据挖掘与分析方法。首先在分析五台山url特征基础上采用布隆过滤器算法实现对五台山重复url的过滤;其次,使用Xpath路径表达式,对五台山网页内容进行提取;最后,通过对内容变长分块操作,同时对文档进行了Bloom Filter编码,实现了网页内容去重操作,进而提取有用信息。With the development of the Internet,the network information has an explosive growth,there is a lot of useful information,butalso full of a lot of repeated information.To solve this problem,a data mining and analysis method based on bloom filter algorithm is proposed.First,based on the analysis of wutai mountain url characteristics,the bloom filter algorithm is adopted to filter the repeated url ofMount Wutai.Secondly,the Xpath path expression is used to extract the content of Mount Wutai web page.Finally,by varying the length of the content into blocks and coding the document at the same time,we realize the de-rework of the web content to extract useful information.

关 键 词：数据挖掘 布隆算法 五台山 网页去重 

分 类 号：TP132[自动化与计算机技术—控制理论与控制工程]