基于链式条件随机场的中文分词改进方法  

AN IMPROVED CHINESE WORD SEGMENTATION METHOD BASED ON CHAIN CONDITIONAL RANDOM FIELDS

作　　者：徐浩煜[1,2,3] 任智慧[1,3] 施俊[3] 周晗[1] 

机构地区：[1]中国科学院上海高等研究院航空通讯技术联合实验室,上海201210 [2]中国科学院大学,北京100049 [3]上海大学通信与信息工程学院,上海200444

出　　处：《计算机应用与软件》2016年第12期211-213,233,共4页Computer Applications and Software

基　　金：国家自然科学基金项目(61471231)

摘　　要：基于链式条件随机场模型的序列标注中文分词方法随着中文分词评测Bakeoff的展开得到广泛应用。词位标注集和特征模板集对该模型的学习至关重要,但当前的研究大多采用单一的标注集和特征模板集进行实验,缺乏标注集和特征模板集结合的尝试,使得中文分词中未登录词识别率不高,从而影响互联网领域语料的分词效果。首次采用六词位标注集结合TMPT-10和TMPT-10`特征模板,并与常见标注集和特征模板集的组合在Bakeoff语料上进行实验对比,结果表明,改进的方法 6tag-tmpt10取得更好的未登录词召回率,在互联网领域中文分词能取得很好的效果;同时在F值上也与其他最好结果相当。Abstract With the development of Chinese word segmentation evaluation Bakeoff, the word-position-based tagging Chinese word segmentation approaches based on chain conditional random fields have been widely used. For the training of CRF models, it is essential to select the tone tag set and feature template. However, the researches in the literature generally used single tag set or feature template, lacking of frequently-used tag sets and feature templates in combination, which resulted in out-of-vocabulary rate at a low level and influenced the performance of word segmentation on Internet corpuses. This method firstly combines six-tag set and feature template TMPT-10 and TMPT-10＇, dealing with comparative experiments with frequently-used tag sets and feature templates on the Bakeoff corpuses. The results demonstrate that the improved method 6tag-tmpt10 can reach higher rate of out-of-vocabulary word recall compared with other methods ,which can improve the performance of Chinese word segmentation in Internet field, in the meanwhile can get the comparative Fl-score.

关 键 词：中文分词 词位标注 条件随机场 特征模板 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]