利用词性信息改进Katz平滑算法  

Improved Katz smoothing algorithms with POS information

作　　者：赵岩[1] 王晓龙[1] 徐志明[1] 刘秉权[1] 

机构地区：[1]哈尔滨工业大学计算机科学与技术学院,哈尔滨150001

出　　处：《哈尔滨工业大学学报》2007年第9期1445-1448,共4页Journal of Harbin Institute of Technology

基　　金：国家自然科学基金重点资助项目(60435020);国家高技术研究发展计划资助项目(2002AA117010-09)

摘　　要：对已有的N-gram平滑算法进行了系统地分析,分别实现了Absolute、W-B和Katz平滑算法.为解决传统Katz平滑算法在处理某些汉语固定搭配时无法进行概率折扣的问题,利用词性信息构造了新的折扣系数.新的折扣系数使词频越大,折扣越小,后接词越多,折扣越大,满足平滑算法对折扣系数的期望.试验结果表明:新的Katz平滑算法降低了N-gram模型的交叉熵,在汉语分词中应用改进的平滑算法也提高了分词结果的F量度.This paper reviewed existing smoothing methods for N - gram model firstly, and implemented the Absolute, W - B and Katz smoothing algorithms respectively. Traditional Katz algorithm couldn＇ t discount the probability when it smoothed Chinese collocation. We constructed new discounting coefficient based on Part-of- Speech information to resolve this problem. Calculated by the new discounting coefficient, discount could decrease when word frequency increased, and the more count of following word, the more discount. All this satisfied demand of smoothing methods. Experiment result showed that improved Katz smoothing algorithm could not only decrease the cross entropy of language model, but also increase the F measure when applied to Chinese word segmentation.

关 键 词：N-GRAM模型 数据稀疏 词性信息 Katz平滑 

分 类 号：TP391.2[自动化与计算机技术—计算机应用技术]