一种基于信息熵的中文高频词抽取算法  

A Chinese Word Extraction Algorithm Based on Information Entropy

作　　者：任禾[1] 曾隽芳[1] 

机构地区：[1]中国科学院自动化研究所综合信息中心,北京100080

出　　处：《中文信息学报》2006年第5期40-43,90,共5页Journal of Chinese Information Processing

摘　　要：为扩展分词词典,提高分词的准确率,本文提出了一种基于信息熵的中文高频词抽取算法,其结果可以用来识别未登录词并扩充现有词典。我们首先对文本进行预处理,将文本中的噪音字和非中文字符转化为分隔符,这样文本就可以被视为用分隔符分开的中文字符串的集合,然后统计这些中文字符串的所有子串的相关频次信息,最后根据这些频次信息计算每一个子串的信息熵来判断其是否为词。实验证明,该算法不仅简单易行,而且可以比较有效地从文本中抽取高频词,可接受率可达到91.68%。Targeting at extending the dictionary for word segmentation so as to improve its accuracy, this paper presents a high-frequency Chinese word extraction algorithm based on information entropy. We firstly transform noisy words and characters to separators, thus a text can be viewed as a Chinese string collection isolated by separators. Then we compute the frequencies of all the substrings of these Chinese strings. Finally, we judge whether each substring is a word by computing its information entropy. Preliminary experiments show that this simple algorithm is effective in extracting high-frequency Chinese words, with the accept rate up to 91.68%.

关 键 词：人工智能 自然语言处理 分词 中文抽词 信息熵 高频词 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]