利用上下文信息解决汉语自动分词中的组合型歧义  

Solving Combinatorial Ambiguity in Chinese Word Segmentation Using Contextual Information

作　　者：肖云[1] 孙茂松[1] 邹嘉彦[2] 

机构地区：[1]清华大学智能技术与系统国家重点实验室,北京100084 [2]香港城市大学语言资讯科学研究中心

出　　处：《计算机工程与应用》2001年第19期87-89,106,共4页Computer Engineering and Applications

基　　金：国家重点基础研究发展规划项目资助课题(课题编号:G1998030507)

摘　　要：组合型歧义切分字段一直是汉语自动分词研究中的一个难点。该文将之视为与WordSenseDisambiguation(WSD)相等价的问题。文章借鉴了WSD研究中广泛使用的向量空间法,选取了20个典型的组合型歧义进行了详尽讨论。提出了根据它们的分布“分而治之”的策略,继而根据实验确定了与特征矩阵相关联的上下文窗口大小、窗口位置区分、权值估计等要素,并且针对数据稀疏问题,利用词的语义代码信息对特征矩阵进行了降维处理,取得了较好的效果。笔者相信,这个模型对组合型歧义切分字段的排歧具有一般性。Combinatorial ambiguity is a vital issue in Chinese word segmentation.We regard it as an equivalence of the problem of word sense disambiguation(WSD)in language computing.In sight of the vector space model commonly used in WSD and based on detailed observations on 20 typical combinatorial ambiguities,this paper at first presents the strategy of treating these ambiguities separately according to their distribution,then determines by experiments the key factors regarding feature matrix(the size of the context window,the sensitivity of locations in the window as well as weighting of feature words),and lastly makes use of semantic codes of words so as to reduce the dimension of the feature matrix.Preliminary results show that the proposed scheme is satisfactory in performance and may serve as a general solution for processing combinatorial ambiguities.

关 键 词：自然语言处理 中文计算 汉语自动分词 组合型歧义切分字段 中文信息处理 

分 类 号：TP391.12[自动化与计算机技术—计算机应用技术]