Vari-gram language model based on word clustering  

作　　者：袁里驰 

机构地区：[1]School of Information Technology,Jiangxi University of Finance and Economics [2]School of Information Science and Engineering,Central South University

出　　处：《Journal of Central South University》2012年第4期1057-1062,共6页中南大学学报（英文版）

基　　金：Project(60763001) supported by the National Natural Science Foundation of China;Project(2010GZS0072) supported by the Natural Science Foundation of Jiangxi Province,China;Project(GJJ12271) supported by the Science and Technology Foundation of Provincial Education Department of Jiangxi Province,China

摘　　要：Category-based statistic language model is an important method to solve the problem of sparse data.But there are two bottlenecks:1) The problem of word clustering.It is hard to find a suitable clustering method with good performance and less computation.2) Class-based method always loses the prediction ability to adapt the text in different domains.In order to solve above problems,a definition of word similarity by utilizing mutual information was presented.Based on word similarity,the definition of word set similarity was given.Experiments show that word clustering algorithm based on similarity is better than conventional greedy clustering method in speed and performance,and the perplexity is reduced from 283 to 218.At the same time,an absolute weighted difference method was presented and was used to construct vari-gram language model which has good prediction ability.The perplexity of vari-gram model is reduced from 234.65 to 219.14 on Chinese corpora,and is reduced from 195.56 to 184.25 on English corpora compared with category-based model.Category-based statistic language model is an important method to solve the problem of sparse data. But there are two bottlenecks： 1） The problem of word clustering. It is hard to find a suitable clustering method with good performance and less computation. 2） Class-based method always loses the prediction ability to adapt the text in different domains. In order to solve above problems, a definition of word similarity by utilizing mutual information was presented. Based on word similarity, the definition of word set similarity was given. Experiments show that word clustering algorithm based on similarity is better than conventional greedy clustering method in speed and performance, and the perplexity is reduced from 283 to 218. At the same time, an absolute weighted difference method was presented and was used to construct vari-gram language model which has good prediction ability. The perplexity of vari-gram model is reduced from 234.65 to 219.14 on Chinese corpora, and is reduced from 195.56 to 184.25 on English corpora compared with category-based model.

关 键 词：word similarity word clustering statistical language model vari-gram language model 

分 类 号：TP311.13[自动化与计算机技术—计算机软件与理论]