基于Counting Bloom Filter的海量网页快速去重研究  

Rapid De-Duplication of Massive Web Page Based on Counting Bloom Filter

作　　者：刘年国[1] 王芬[1] 吴家奇[1] 李雪[1] 陶涛[1] 

机构地区：[1]国网淮南供电公司,安徽淮南232007

出　　处：《安徽电气工程职业技术学院学报》2016年第3期92-97,共6页Journal of Anhui Electrical Engineering Professional Technique College

摘　　要：网页去重是从给定的大量的数据集合中检测出冗余的网页,然后将冗余的网页从该数据集合中去除的过程,其中基于同源网页的URL去重的研究已经取得了很大的发展,但是针对海量网页去重问题,目前还没有很好的解决方案,文章在基于MD5指纹库网页去重算法的基础上,结合Counting Bloom Filter算法的特性,提出了一种快速去重算法IMP-CBFilter。该算法通过减少I/O频繁操作,来提高海量网页去重的效率。实验表明,IMP-CBFilter算法的有效性。Web page de-duplication is a process which detects redundant duplicate content pages from a given amount of data collection, and then removes from the collection. The research of web de-duplica- tion based on URL filter has achieved great development. But there is no ideal solution to solve this kind of problem with massive web pages fiher. Based on MD5 fingerprint database web de-duplication algo- rithm, and the utilization of Counting Bloom filter algorithm, an algorithm for rapid de-duplication named as IMP-CBFiher has been proposed in this paper. It could improve the efficiency of mass web pages filter by reducing the frequent operation of I/O. The results indicate higher performance by using IMP-CBFilter algorithm.

关 键 词：网页去重 MD5指纹库 COUNTING BLOOM Filter IMP-CBFilter算法 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]