汉语交集型歧义切分字段关于专业领域的统计特性  

Statistical Properties of Overlapping Word Segmentation Ambiguities in Domain-specific Chinese Corpora

作　　者：乔维[1] 孙茂松[1] 

机构地区：[1]清华信息科学与技术国家实验室(筹),清华大学计算机科学与技术系,北京100084

出　　处：《中文信息学报》2008年第4期10-18,共9页Journal of Chinese Information Processing

基　　金：国家自然科学基金资助项目(60573187);国家863计划资助项目(2007AA01Z148)

摘　　要：交集型分词歧义是汉语自动分词中的主要歧义类型之一。现有的汉语自动分词系统对它的处理能力尚不能完全令人满意。针对交集型分词歧义,基于通用语料库的考察目前已有不少,但还没有基于专业领域语料库的相关考察。根据一个中等规模的汉语通用词表、一个规模约为9亿字的通用语料库和两个涵盖55个专业领域、总规模约为1.4亿字的专业领域语料库,对从通用语料库中抽取的高频交集型歧义切分字段在专业领域语料库中的统计特性,以及从专业领域语料库中抽取的交集型歧义切分字段关于专业领域的统计特性进行了穷尽式、多角度的考察。给出的观察结果对设计面向专业领域的汉语自动分词算法具有一定的参考价值。Overlapping ambiguity is a major type of ambiguity in Chinese word segmentation. The performance of existing word segmentation systems in resolving this ambiguity is still unsatisfactory, especially in dealing with domain-specific texts. In contrast to those quite detailed statistical observations on overlapping ambiguities in generalpurpose corpus, similar observations in domain-specific corpus have not been reported in the literature. In terms of a medium-sized general-purpose Chinese wordlist, a general-purpose corpus with over 900 million characters and two domain-specific corpora with total 140 million characters covering 55 domains, statistical properties of high frequent overlapping ambiguities are addressed and studied from several perspectives： with overlapping ambiguity string from general corpus examined in the domain corpus, and vice versa. It is believed that the finding of this paper will benefit word segmentation disambiguation in particular for domain-specific texts.

关 键 词：计算机应用 中文信息处理 汉语自动分词 专业领域语料库 交集型歧义切分字段 伪歧义 真歧义 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]