基于多特征的自适应新词识别  

Adaptive Method for Chinese New Word Identification Based on Multi-features

作　　者：罗智勇[1] 宋柔[2] 

机构地区：[1]北京工业大学计算机学院,北京100022 [2]北京语言大学语言信息处理研究所,北京100083

出　　处：《北京工业大学学报》2007年第7期718-725,共8页Journal of Beijing University of Technology

基　　金：国家自然科学基金(60272055;60572159);国家'八六三'计划资助项目(2001AA114111);教育部科学技术研究重点项目(00128;107017).

摘　　要：为提高自动分词系统对未登录词的识别性能,提出和实现了一种基于多特征的自适应新词识别方法,综合考虑了被处理文本中重复字符串的上下文统计特征(上下文熵)、内部耦合特征(似然比)、背景语料库对比特征(相关频率比值)以及自动分词系统辅助的边界确认信息等,并直接从被抽取文本中自动训练识別模型．同时,新词识别过程在字串PAT-Array数据结构上进行,可以抽取任意长度的新词语．实验结果表明,该方法新词发现速度快、节省存储空间．To improve the performance of new word identification in Chinese word segment, the authors propose an adaptive method for Chinese new word identification based on multi-feature method for offline corpus processing, in which many features, including context-entropy, likelihood ratios, frequency ratio against background corpus and boundary-verification with basic segmentation are introduced to evaluate the candidate words. And all of the features are integrated into an adaptive SVM classifier. Candidate new words are extracted efficiently on PAT-Array with much less space overhead and arbitrary n-gram words can be identified by the method. The results show that the method can run fast upon new word identification and save much memory.

关 键 词：自然语言处理系统 计算语言学 词语处理 新词识别 多特征 自适应 自动分词 

分 类 号：TP391.12[自动化与计算机技术—计算机应用技术]