基于相关性及语义的n-grams特征加权算法  

n-grams Features Weighting Algorithm Based on Relevance and Semantic

作　　者：邱云飞[1] 刘世兴[1] 林明明[1] 邵良杉[2] 

机构地区：[1]辽宁工程技术大学软件学院,葫芦岛125105 [2]辽宁工程技术大学系统工程研究所,葫芦岛125105

出　　处：《模式识别与人工智能》2015年第11期992-1001,共10页Pattern Recognition and Artificial Intelligence

基　　金：国家自然科学基金项目(No.70971059);辽宁省创新团队项目(No.2009T045);辽宁省高等学校杰出青年学者成长计划项目(No.LJQ2012027)资助

摘　　要：n-grams作为文本分类特征时易造成分类准确率下降,并且在对n-grams加权时通常忽略单词间的冗余度和相关性.针对上述问题,文中提出基于相关性及语义的n-grams特征加权算法.在文本预处理时,对n-grams进行特征约简,降低内部冗余,再根据n-grams内单词与类别的相关性及n-grams与测试集的语义近似度加权.搜狗中文新闻语料库和网易文本分类语料库上的实验表明,文中算法能筛选高类别相关且低冗余的n-grams特征,在量化测试集时减少稀疏数据的产生.When n-grams are considered as text classification features, the classification accuracy is decreased. The redundancy and relevance between words are ignored while n-grams are weighted. Thus, n-grams features weighting algorithm based on relevance and semantic is proposed. To decrease the internal redundancy, feature reduction is conducted to n-grams during text preprocessing. Then, n-grams are weighted according to the relevance of words and classes in n-grams and the semantic similarity of n-grams and testing dataset. The experimental results on Sougo Chinese news corpse and NetEase text corpse show that the proposed algorithm can select n-grams features of high relevance and low redundancy, and reduce the sparse data while quantifying the testing dataset.

关 键 词：最大相关度最小冗余度(mRMR) 语义相似度 N-GRAMS 特征加权 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]