一种基于层叠CRF的古文断句与句读标记方法  

Method of sentence segmentation and punctuating for ancient Chinese literatures based on cascaded CRF

作　　者：张合[1] 王晓东[1] 杨建宇 周卫东[3] 

机构地区：[1]河南师范大学计算机与信息技术学院,河南新乡453007 [2]北京得意音通技术有限责任公司,北京100085 [3]北京交通大学信息所,北京100044

出　　处：《计算机应用研究》2009年第9期3326-3329,共4页Application Research of Computers

基　　金：河南省科技厅攻关资助项目(0624480021)

摘　　要：针对利用自然语言理解技术进行古汉语断句及句读标注的主要挑战是数据稀疏问题,设计了一种六字位标记集,提出了一种基于层叠式CRF模型的古文断句与句读标记方法。基于六字位标集,低层模型用观察序列确定句子边界,高层模型同时使用观察序列和低层的句子边界信息进行句读标记。实验在5M混合古文语料上分别进行了封闭测试和开放测试,封闭测试断句与句读标注的F值分别达到96.48%和91.35%,开放测试断句与句读标注的F值分别达到71.42%和67.67%。Data sparseness is a primary challenge in sentence segmentation and punctuating for ancient Chinese literatures using natural language processing technology. In order to overcome this difficulty, designed a 6-tag set and proposed a method based on cascaded conditional random fields. The main idea was as follows ： basing on the 6-tag set, a low level model deter- mined the boundaries of sentences according to observation sequence and a high level model punctuated sentences taking con- sideration of both observation sequence and low level＇ s results. Done close test and open test based on approximate 5M mixed corpus respectively. The F measure of sentence segmentation and punctuation were 96.48%. and 91.35% respectively in close test, and those were 71.42% and 67.67% respectively in open test.

关 键 词：古汉语 层叠条件随机场 数据稀疏 断句 句读标注 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]