基于字词分类的层次分词方法  

Method of Chinese word segmentation based on character-word classification

作　　者：张聪品[1] 赵理莉[1] 吴长茂[1] 

机构地区：[1]河南师范大学计算机与信息技术学院,河南新乡453007

出　　处：《计算机应用》2010年第8期2034-2037,共4页journal of Computer Applications

基　　金：河南省基础与前沿技术研究计划项目(092300410152);河南省高等学校青年骨干教师资助计划项目;河南省科技攻关项目(2007520026)

摘　　要：中文分词是自然语言处理的基础性问题。条件随机场模型分词过程中出现的切分粒度过小和多字粘连造成的错分问题,是影响分词结果的两个主要原因。提出了一个基于字词分类的层次分词模型,该模型采用多部有效词典进行处理,在外层分词系统中解决切分粒度过小问题;在内层核心层,条件随机场分词后再处理多字粘连问题。实验结果表明,采用加入多词典的字词结合层次分类模型F-测度值有较大的提高,有助于得到好的分词结果。Chinese Word Segmentation CWS is the basic problem of natural language processing.A joint characterword classification model for Chinese Word Segmentation was presented which mainly dealt with the problem of Conditional Random Field CRF.On the one hand the majority of errors in CRF caused by fine granularity were figured out in outsidelayer of the model.On the other hand excessive linked words caused by improper segmentations that were settled in core inside-layer.The experimental results show that the value of F is greatly improved and the good results of word segmentation are easily gained by choosing the hierarchical word segmentation model based on character word classification.

关 键 词：中文分词 字词分类 多词典分词 条件随机场 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]