中文高频词串的抽取及其在语言模型中的应用  

Chinese Frequent String Extraction and Application on Language Model

作　　者：文娟[1] 王小捷[1] 

机构地区：[1]北京邮电大学智能科学技术研究中心,北京100876

出　　处：《北京邮电大学学报》2009年第5期10-14,共5页Journal of Beijing University of Posts and Telecommunications

基　　金：国家科技支撑计划项目(2007BAH05B02-04);高等学校学科创新引智计划项目(B08004);BUPT-Nokia合作项目

摘　　要：为准确抽取语料库中的高频词串,使其能更好地应用于语言模型中,提出了一种基于字串切分度的中文高频词串(CFS)抽取算法,并用该算法抽取出的CFS分别建立一元和二元语言模型.实验表明,基于CFS的语言模型能有效克服现有基于字和词的n元语法模型长距离相依性能较差的缺陷;同时,在模型困惑度、音字转换正确率上均优于已有基于净频次的CFS语言模型.In order to extract the Chinese frequent strings（CFS） accurately and make better use in language models,a new method for CFS extraction using string segmentation degree is proposed.Unigram and bigram language models based on this CFS extraction method are built.Experiment shows that the CFS based language model can deal with the lack of long distance dependency problem in character and word based language model.It also shows that the CFS based language model has lower model perplexity and higher pinyin-to- character conversion correctness compared with the model based on previous CFS extraction method.

关 键 词：中文高频词串 字区分度 字串切分度 N元模型 音字转换 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]