一种双层网页去重方法研究  

Research on Deletion of Duplicated Web Pages on Two Levels

作　　者：毛晓蛟[1] 

机构地区：[1]南京师范大学强化培养学院,南京210046

出　　处：《电脑编程技巧与维护》2010年第20期66-67,84,共3页Computer Programming Skills & Maintenance

摘　　要：利用Bloom Filter数据结构、shingling算法和MD5编码,构造双层网页去重模型。通过Bloom Filter结构,在网络蜘蛛程序下载网页时,去除重复的网址,并讨论了Bloom Filter出错概率。对已下载的网页用shingling算法去重,阐述了相似网页的判断方法。通过实验,得到了最后的结果,并指出了模型存在的缺点和该方法的后续研究方向。This paper constructs the model of deletion of Duplicated web collections on two levels with Bloom Filter、Shingling Algorithm and MD5. With the help of Bloom Filter, it deletes Duplicated web collections while the web Spider is working. And also discuss the false rate of Bloom Filter. Then using Shingling to judge similar web pages and delete similar ones. Get the final results through experiments and put forward directions of further study.

关 键 词：BLOOM FILTER 错误率 shingling MD5 相似网页 

分 类 号：TP393.092[自动化与计算机技术—计算机应用技术]