基于后缀树的中文新闻重复网页识别算法  

An Algorithm for Detecting Duplicated Chinese Web News Based on Suffix Tree

作　　者：钱爱兵[1] 江岚[1] 

机构地区：[1]南京大学信息管理系,南京210093

出　　处：《现代图书情报技术》2008年第3期55-61,共7页New Technology of Library and Information Service

摘　　要：针对识别中文新闻重复网页传统方法的不足,提出以后缀树作为基本数据结构,依据新闻网页的标题性和时间性,构建中文新闻重复网页识别算法。该算法以Ukkonen算法和Matching Statistics算法为基础,并对其具体实现进行优化。实验结果表明,该算法不仅具有有效性,而且对计算字符串相似度也有启发意义。In view of the shortcomings of traditional methods detecting duplicated Chinese Web news, this paper proposes the suffix tree as the basic data structure, and then designs an algorithm for detecting duplicated Chinese Web news according to the title and published date of news. This algorithm is based on Ukkonen algorithm and Matching Statistics algorithm and optimizes the realization. Experiments show that the proposed algorithm is an effective solution to detect the duplicated Chinese Web news, and it can be generalized to measure the similarity of strings.

关 键 词：后缀树 重复网页 Ukkonen算法 匹配统计算法 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]