基于改进最大匹配算法的中文分词粗分方法  

Method of Chinese words rough segmentation based on improving maximum match algorithm

作　　者：周俊[1,2] 郑中华[3] 张炜 

机构地区：[1]华中科技大学模具技术国家重点实验室,武汉430074 [2]安徽博约信息科技有限责任公司,合肥230088 [3]中国人民大学教育学院,北京100872

出　　处：《计算机工程与应用》2014年第2期124-128,共5页Computer Engineering and Applications

基　　金：国家自然科学基金(No.70773052)

摘　　要：中文粗分和歧义消解是中文分词的两大基本过程。通过引入广义词条和诱导词集,在最大匹配算法基础上提出一种中文分词的粗分方法,以最长广义词匹配为原则进行中文分词,利用诱导词集实现交叉型歧义识别。在保证快速准确切分无歧义汉语语句的同时,100%检测并标记有歧义汉语语句中的交叉型歧义,最大程度上简化后续歧义消解过程。通过对含有160万汉字1998年1月人民日报语料测试的结果证明了算法速度、歧义词准确率以及粗分召回率的有效性。Chinese words rough segmentation and ambiguity resolution are two fundamental processes of Chinese word segmentation. Under the introduction of generalized term and induced word set, a method used for Chinese words rough segmentation is proposed based on maximum matching method. It executes Chinese word segmentation under the principle of the longest generalized term matching, and recognizes the overlapping ambiguities by utilizing induced word set. It seg- ments Chinese sentences without any ambiguity rapidly and accurately, detects and marks ambiguities by 100 percent in those sentences which have ambiguities, which will simplify the process of ambiguity resolution to the maximum extent. The result of the experiment on People＇ s Daily corpus in January 1998 which contains 1.6 million Chinese characters shows the method is effective both in speed and accuracy.

关 键 词：中文分词 最大匹配 广义词 诱导词集 

分 类 号：TP391.12[自动化与计算机技术—计算机应用技术]