中文分词任务中标注集合的选择方法  

Selection Method of Tagging sets for Chinese Segmentation Task

作　　者：宋毅君[1] 王瑞波[1] 史立校 

机构地区：[1]山西大学现代教育技术中心,太原030006

出　　处：《山西大学学报（自然科学版）》2016年第2期204-209,共6页Journal of Shanxi University(Natural Science Edition)

基　　金：国家自然科学基金(60873128)

摘　　要：统计分词模型需要使用合适的标注集合来将中文句子的分词问题转化成序列标注问题,因此,标注集合的选择是统计分词模型中需要解决的一个重要的问题。文章分别借助单因子方差分析、置信区间和p值三种方法来比较常用的几种标注集合对中文分词模型性能差异的显著程度。首先根据单因子方差分析方法得出标注集合对于中文分词模型性能的影响是显著的;其次,使用置信区间和p值来选择最优的标注集合。实验结果表明:在常用的标注集合中,标注集合BB2B3MSE可以明显地提升中文分词模型的性能。Statistical Chinese word segmentation models are usually casted as a sequential labeling problem,and a proper tagging set is provided as an important prerequisite condition,therefore,selection method for the proper tagging set is an important problem to be solved.We employ three methods of one-way ANOVA,confidential interval and p-value to compare several widely-used tagging sets.We firstly using oneway ANOVA to demonstrate that selection of tagging sets significantly affect performance of statistical Chinese word segmentation models.And then,we select the best tagging set through methods of confidential interval and p-value.The experiment results illustrates that BB2B3 MSE as a best tagging set can improve the performance of Chinese segmentation models obviously.

关 键 词：中文分词 条件随机场 标注集合 F-测度 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]