最大熵和规则相结合的藏文句子边界识别方法  

A Maximum Entropy and Rules Approach to Identifying Tibetan Sentence Boundaries

作　　者：李响[1] 才藏太[2] 姜文斌[1] 吕雅娟[1] 刘群[1] 

机构地区：[1]中国科学院计算技术研究所,中国科学院智能信息处理重点实验室,北京100190 [2]青海师范大学计算机学院,青海西宁810008

出　　处：《中文信息学报》2011年第4期39-44,共6页Journal of Chinese Information Processing

基　　金：国家自然科学基金重大研究计划培育项目(90920004),国家自然科学基金重点资助项目(60736014)

摘　　要：句子边界识别是藏文信息处理领域中一项重要的基础性工作,该文提出了一种基于最大熵和规则相结合的方法识别藏语句子边界。首先,利用藏语边界词表识别歧义的句子边界,最后采用最大熵模型识别规则无法识别的歧义句子边界。该方法有效利用藏语句子边界规则减少了最大熵模型因训练语料稀疏或低劣而导致对句子边界的误判。实验表明,该文提出的方法具有较好的性能,F1值可达97.78%。Sentence boundary identification is a fundamental work in the field of Tibetan information processing.This paper proposes a maximum entropy and rules approach to identifying Tibetan sentence boundaries.First,the Tibetan boundary vocabulary based detector identifies the ambiguous sentence boundaries.Second,the maximum entropy model based detector identifies the ambiguous sentence boundaries which the former detector can＇t identify.By making use of Tibetan sentence boundary rules,this approach further reduces the number of the incorrect sentence boundary identified by maximum entropy model owing to the sparse and inferior training corpus.The experiments show that this approach has a good performance in terms of 97.78% F1-measure.

关 键 词：最大熵 句子边界识别 藏文信息处理 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]