基于网页内容的网页消重高效检测算法研究  

High-efficiency Detective Algorithm Research for Web-Page-Content-Based Duplication Elimination

作　　者：王祖析[1] 

机构地区：[1]湖南化工职业技术学院,湖南株洲412004

出　　处：《佳木斯大学学报（自然科学版）》2010年第1期22-24,共3页Journal of Jiamusi University：Natural Science Edition

摘　　要：在对现有主流网页消重技术分析的基础上,提出一种基于网页内容的改进的网页消重高效检测算法.该算法通过利用网页的标签树结构选取最大的多个文本块,将这些文本块连接在一起生成一个代表该网页的MD5指纹,对指纹进行比较,确认近似网页实现消重,实验证明该方法对近似网页能进行准确的检测.Based on technical mainstream elimination of duplicated web pages analysis, a high - efficiency and improved detective Algorithm was presented for web - page - content- based duplication elimination. The algorithm selected the maximum number of text blocks by using the web page tag tree structure, and connected these blocks together to generate a web page with typical MD5 fingerprint, to compare the fingerprints, and then to confirm similar web pages to eliminate the duplications. It is proved by experiments that this Algorithm can accurately detect similar web pages.

关 键 词：搜索引擎 网页消重 MD5指纹 算法分析 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]