一种基于EM非监督训练的自组织分词歧义解决方案  

A Self-organized Scheme for Word Segmentation Ambiguity Resolution Based on EM Training Algorithm

作　　者：王伟[1] 钟义信[1] 孙建[1] 杨力[1] 

机构地区：[1]北京邮电大学智能中心,181北京100876

出　　处：《中文信息学报》2001年第2期38-44,共7页Journal of Chinese Information Processing

基　　金：国家自然科学基金资助! (6 9982 0 1)

摘　　要：本文旨在提供一种基于非监督训练的分词歧义解决方案和一种分词算法。基于EM的思想 ,每个句子所对应的所有 (或一定范围内 )的分词结果构成训练集 ,通过这个训练集和初始的语言模型可以估计出一个新的语言模型。最终的语言模型通过多次迭代而得到。通过一种基于该最终语言模型的统计分词算法 ,对于每个句子至少带有一个歧义的测试集的正确切分精度达到 85 .36 % (以句子为单位 )This paper is mainly to present a word segmentation ambiguity resolution scheme based on unsupervised training.According to the idea of EM,a language model is built increasingly by collection the fractional counts of patterns (such as bigram pair)from the augmentations of all the segmentation candidates of a sentence.The learned language model is incorporated into a statistical segmentor.Experiments show that this scheme can resolve 85.36% ambiguity on test set each sentence of which has at least one ambiguous part(and the accuracy rate is based on sentence).

关 键 词：EM算法 分词歧义 非监督训练 分词语言模型 歧义消除 汉语处理 训练算法 分词算法 

分 类 号：TP391.12[自动化与计算机技术—计算机应用技术]