一种基于字和子串联合标注的汉语分词方法  

Combining Character-Based and Substring-Based Tagging for Chinese Word Segmentation

作　　者：于江德[1] 谷川[1] 葛文英[1] 樊孝忠[2] 

机构地区：[1]安阳师范学院计算机与信息工程学院,河南安阳455002 [2]北京理工大学计算机科学技术学院,北京100081

出　　处：《山西大学学报（自然科学版）》2011年第3期357-362,共6页Journal of Shanxi University(Natural Science Edition)

基　　金：高等学校博士学科点专项科研基金(20050007023);河南省高等学校青年骨干教师项目(2009GGJS-108)

摘　　要：近年来基于字的词位标注的方法极大地提高了汉语分词的性能,该方法将汉语分词转化为字的词位标注问题,借助于优秀的序列数据标注模型,基于字的词位标注汉语分词方法逐渐成为汉语分词的主要技术路线.针对一些领域文本中含有较多的英文词汇、缩写、数字等非汉字子串,提出了一种基于字和子串联合标注的汉语分词方法,该方法将子串看作和汉字等同的一个整体,采用四词位标注集,使用条件随机场模型深入研究了基于字和子串联合标注的汉语分词技术.在CIPS-SIGHAN2010汉语分词评测所提供的文学、计算机、医药、金融四个领域语料上进行了封闭测试,实验结果表明此方法比传统的字标注分词方法性能更好.The performance of Chinese word segmentation has been greatly improved by word-position-based approaches in recent years.This approach treats Chinese word segmentation as a word-position tagging problem.With the help of powerful sequence tagging model,word-position-based method quickly rose as a mainstream technique in this field.Considering plenty of abbreviations,numeric and other non-Chinese strings,our segmentation system adopted a method combining character-based and substring-based tagging.Substring is considered a whole in this method.We further studied this technique by using four word-positions and conditional random fields.Closed evaluations are performed on four corpuses from the CIPS-SIGHAN 2010,namely Literary,Computer,Medicine and Finance.And comparative experiments are performed.Experimental results show that the method is much better performance than the traditional character-based tagging method.

关 键 词：汉语分词 条件随机场 词位标注 子串 特征模板 

分 类 号：N55[自然科学总论]