基于文本相似度的网页消重策略  

STRATEGY OF ELIMINATING DUPLICATED WEB PAGES BASED ON TEXT SIMILARITY

作　　者：刘书一[1] 

机构地区：[1]江苏科技大学南徐学院,江苏镇江212013

出　　处：《计算机应用与软件》2011年第11期228-229,278,共3页Computer Applications and Software

摘　　要：针对在网页检索结果中经常出现内容相同或相似的问题,提出了一种通过计算网页相似度的方法进行网页消重。该算法通过提取网页特征串,特征串的提取在参考以往特征码提取的基础上,加入了文本结构特征的提取,通过比较特征串之间差异性的基础上得到网页的相似度。经与相似方法比较,结果表明,该方法减少了时间复杂度,具有较高的查全率和查准率,适于大规模网页消重。In order to solve the problems of identical or similar contents frequently occur in web pages retrieval, a method of eliminating duplicated web pages by calculating the similarity of the web pages was put forward. This algorithm uses the extraction of feature string of the web pages. Such extraction of feature string is based on previous feature code extraction as the reference and is combined with the extraction of text structural features. By comparing differences between the feature strings the similarities of web pages are derived. The result of comparing it with other similar algorithms reveals that this method has higher recall and precision ratio and reduces the time complexity, and is suitable for eliminating large-scale duplicated web pages.

关 键 词：网页消重 网页特征码 文本相似度 编辑距离 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]