基于混合语言模型的文档相似性计算模型  

Document Similarity Model Based on Mixture Language Model

作　　者：李晓光[1] 于戈[1] 王大玲[1] 

机构地区：[1]东北大学信息科学与工程学院,辽宁沈阳110004

出　　处：《中文信息学报》2006年第4期41-48,共8页Journal of Chinese Information Processing

基　　金：国家自然科学基金资助项目(60573090;60503036;60473073)

摘　　要：为了克服现有文档相似性模型对文档特性拟合的不完全性和缺乏理论根据的弱点,本文在统计语言模型的基础上,提出了一种基于混合语言模型(M ixture Language Model,MLM)文档相似性计算模型。MLM利用统计语言模型描述文档特征,将相关影响因素作为模型的潜在子模型,文档语言模型由各子模型混合构成,从而准确和全面地反映文档特征。由于MLM根据具体应用确定相关影响因素,并以此构建相应文档描述模型,因此具有很强的灵活性和扩展性。在MLM的基础上,本文给出了一个基于文档主题内容相似性的实例,在TREC9数据集上的实验表明MLM优于向量空间模型(VSM)。To overcome the incompleteness of modeling document characteristics and the lack of theory for current document similarity models, this paper puts forward to utilize mixture language model （MLM） to evaluate document-to- document similarity. In MLM, the characteristic of a document is described based on statistic language model, and the factors of influencing its characteristic are viewed as the latent models, and then the document language model is a mixture model combined with each latent models. MLM not only models document characteristics more perfectly, but it is flexible and scalable to be implemented with respect to applications. Within the framework of MLM, a document similarity method is presented from the viewpoint of document content. The experimental results over the TREC9 dataset indicate that MLM outperforms VSM.

关 键 词：人工智能 自然语言处理 文档相似性 统计语言模型 混合模型 EM算法 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]