基于改进互信息和邻接熵的微博新词发现方法  

Micro-blog new word discovery method based on improved mutual information and branch entropy

作　　者：夭荣朋 许国艳[1] 宋健[1] 

机构地区：[1]河海大学计算机与信息学院,南京211100

出　　处：《计算机应用》2016年第10期2772-2776,共5页journal of Computer Applications

基　　金：国家科技支撑计划项目(2013BAB06B04);江苏省自然科学基金资助项目(BK20130852);江苏水利科技项目(2013025);中国华能集团公司总部科技项目(HNKJ13-H17-04)~~

摘　　要：针对目前微博新词发现算法中的数据稀疏、可移植性较差以及缺乏对多字词(大于三字)识别的问题,提出了基于改进互信息(MI)和邻接熵(BE)的微博新词发现算法——MBN-Gram。首先,利用N元递增算法(N-Gram)提取新词的候选项,对提取出来的候选新词使用频率和停用字等规则进行过滤;接着再利用改进MI和BE对候选项进行扩展及再过滤;最后,结合相应词典进行筛选,从而得到新词。通过理论及实验分析,MBN-Gram算法在准确率、召回率及F值上均有一定提高。实验结果表明,MBN-Gram算法是有效可行的。Aiming at the problem of data sparsity, poor portability and lack of recognition of multiple words （more than three words） in micro-blog new word discovery algorithm, a new word discovery algorithm based on improved Mutual Information （MI） and Branch Entropy （BE）, named MBN-Gram, was proposed. Firstly, the N-Gram was used to extract the candidate terms of new words, and the rules of using frequency and stop words were used to filter the candidates. Then the improved MI and BE were used to expand and filter the candidates again. Finally, the corresponding dictionary was used to screen, so as to get new words. Theoretical and experimental analysis show that the accuracy rate, recall rate and F value of MBN-Gram algorithm were improved. Experimental results shows that the MBN-Gram algorithm is effective and feasible.

关 键 词：新词发现 多字词 N-GRAM 互信息 邻接熵 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]