基于无指导学习策略的无词表条件下的汉语自动分词  

Chinese Word Segmentation without Using Dictionary Based on Unsupervised Learning Strategy

作　　者：孙茂松[1] 肖明[1] 邹嘉彦[2] 

机构地区：[1]清华大学智能技术与系统国家重点实验室,北京100084 [2]香港城市大学语言资讯科学研究中心

出　　处：《计算机学报》2004年第6期736-742,共7页Chinese Journal of Computers

基　　金：国家自然科学基金 ( 60 0 83 0 0 5 )资助

摘　　要：探讨了基于无指导学习策略和无词表条件下的汉语自动分词方法 ,以期对研制开放环境下健壮的分词系统有所裨益 .全部分词知识源自从生语料库中自动获得的汉字Bigram .在字间互信息和t 测试差的基础上 ,提出了一种将两者线性叠加的新的统计量md ,并引入了峰和谷的概念 ,进而设计了相应的分词算法 .大规模开放测试结果显示 ,该算法关于字间位置的分词正确率为 85 .88% ,较单独使用互信息或t 测试差分别提高了 2 4 7%和 5 6 6 % .The study of word segmentation using neither a dictionary nor an annotated corpus will help to improve the robustness of Chinese word segmentation systems in practical applications. This paper presents a new statistical measure, md, for estimating the associative strength between two adjacent Chinese characters in terms of a linear combination of two common statistical measures, namely, mutual information and t-test, puts forward the concepts of peak and valley of md, and gives an algorithm accordingly. The experimental result on large-scale open tests shows that the proposed algorithm achieves 85.88% segmentation accuracy (defined as the ratio of the number of correctly segmented character locations to the total number of character locations in the testing texts). The most advantage of the algorithm is that it can easily adapt to new application domains because all statistical data needed is based on Bigram of Chinese characters derived from raw corpora automatically.

关 键 词：无指导学习 汉语自动分词 汉字Bigram 互信息 t-测试差 线性叠加 统计量 中文信息处理应用系统 

分 类 号：TP391.12[自动化与计算机技术—计算机应用技术]