基于特征迭代的短文本去重算法  

Short Text Duplicate Removal Algorithm Based on Feature Iteration

作　　者：曹海[1] 孙婧[1] 史喜斌[1] 

机构地区：[1]复旦大学计算机科学技术学院上海市数据科学重点实验室,上海201203

出　　处：《计算机工程》2015年第12期54-57,63,共5页Computer Engineering

基　　金：国家科技支撑计划基金资助项目(2012BAH13F02);上海市科委基金资助项目(12511502403;12511509602)

摘　　要：由于短文本具有词频单一、结构简单等特点,基于传统特征选取方法的文本去重算法不适合短文本。为此,提出一种适合短文本特点的去重算法,利用SimHash算法产生短文本的指纹,使用共享最近邻算法对指纹进行聚类,根据聚类结果增删初始特征,迭代直至收敛,从而实现短文本的去重检测。在真实数据集上的实验结果表明,与现有的文本去重算法相比,该算法对于短文本具有更好的去重效果。Because of the single word frequency and the simple structure of short text,algorithms based on normal feature selection methods do not fit to short text.This paper proposes an iteration method of weighting features for short text.It produces the fingerprints of short text using SimHash,and clusters these fingerprints with Shared Nearest Neighbor（SNN）.Initial features are added or deleted according to the clusters.This process is circulatory so as to realize the duplicate removal of short text.Experimental results based on two real datasets show that this method fits short text well and has better duplicate removal effects than existing methods.

关 键 词：SimHash算法 共享最近邻 迭代 特征选择 短文本 去重 

分 类 号：TP311[自动化与计算机技术—计算机软件与理论]