基于LDA模型的文本相似度研究  

Study on Text Similarity Based on LDA Model

作　　者：陈攀[1] 杨浩[1] 吕品[1,2] 王海晖[1,2] 

机构地区：[1]武汉工程大学计算机科学与工程学院,湖北武汉430073 [2]武汉工程大学智能机器人湖北省重点实验室,湖北武汉430073

出　　处：《计算机技术与发展》2016年第4期82-85,89,共5页Computer Technology and Development

基　　金：湖北省高等学校优秀中青年团队计划项目(T201206);湖北省智能机器人重点实验室开放基金(HBIR201409)

摘　　要：LDA主题模型是近年来提出的一种具有文本表示能力的非监督学习模型。考虑到传统主题模型在处理大规模文本时存在的局限性,文中提出一种基于LDA模型的文本相似度计算方法。利用LDA为语料库建模,通过Gibbs抽样间接估算模型参数,将文本表示为固定隐含主题集上的概率分布,以此计算文本之间的相似度。最后将K-means算法作为文本相似度的评估指标。实验结果表明,与LSI模型相比,该方法能有效地提高文本相似度计算的准确性和文本聚类效果。LDA topic model is an unsupervised model which exhibits superiority on latent topic modeling of text data in the research of recent years. Considering the disadvantage of the traditional topic model when dealing with the large- scale text corpuses,a method which improves text similarity computations by using LDA model is proposed. It models corpus with LDA,parameters are estimated with Gibbs sampling. Each document is represented for the probability distribution of fixed implied theme set and computed the similarity between the texts. Finally,the K- means algorithm is selected as the evaluation index of text similarity. Experimental results showthis method can improve the accuracy of text similarity and clustering quality of text effectively compared with LSI model.

关 键 词：文本挖掘 LDA模型 GIBBS抽样 文本相似度 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]