一种改善的基于语言模型的中文检索系统研究  

An Improved Language Model-based Chinese IR System

作　　者：张俊林[1] 曲为民[1] 孙乐[1] 孙玉芳[1] 

机构地区：[1]中科院软件所系统软件与中文信息中心,北京100080

出　　处：《中文信息学报》2004年第2期23-29,43,共8页Journal of Chinese Information Processing

基　　金：国家自然科学基金资助项目 (6 0 2 0 30 0 7) ;国家"十五"86 3重大项目资助 (2 0 0 1AA114 0 4 0 )

摘　　要：最近几年提出的语言模型检索系统将语音识别领域的语言模型技术引入信息检索领域并改善了检索系统的性能 ,但是其隐含的词汇间相互独立的假设并不符合实际情况。尽管统计翻译模型考虑了词汇间的同义词因素 ,但是由于它没有考虑词汇上下文信息 ,所以对于解决多义词词义的区分并无帮助。我们提出了触发语言模型检索方法来改善这一状况 ,通过训练语料得到词汇在一定上下文中的相关比率 ,同时利用查询条件所含词汇计算触发词汇集合来区别查询条件词汇的具体含义并将相关参数引入文档语言模型形成触发语言模型。实验结果表明我们提出的这个方法显著改善了检索系统的性能 ,与经典语言模型方法相比 ,触发语言模型方法的平均查准率提高了约 12 %,召回率提高了 10 8%。Language model based IR system proposed in recent 5 years has introduced the language model approach in the speech recognition area into the IR community and improves the performance of the IR system effectively. However, the assumption that all the indexed words are irrelative behind the method is not the truth. Though statistical MT approach alleviates the situation by taking the synonymy factor into account, it never helps to judge the different meanings of the same word in varied context. In this paper we propose the trigger language model based IR system to resolve the problem. Firstly we compute the association ratio of the words from training corpus and then get the triggered words collection of the query words to find the real meaning of the word in specific text context. We introduce the relative parameters into the document language model to form the trigger language model based IR system. Experiments have shown that the performance of trigger language model based IR system has been improved greatly. Compared with classical language model IR system, Precision of the trigger language model based IR system increased almost 12% and recall of the system increased 10.8%.

关 键 词：计算机应用 中文信息处理 语言模型 信息检索 触发 中文检索系统 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]