基于语义关联和信息增益的TFIDF改进算法研究  

Improved TFIDF feature extraction algorithm based on semantic association and information gain

作　　者：许珂[1] 蒙祖强[1] 林啓峰[1] 

机构地区：[1]广西大学计算机与电子信息学院,南宁530004

出　　处：《计算机应用研究》2012年第2期557-560,共4页Application Research of Computers

基　　金：国家自然科学基金资助项目(61063032);广西教育厅科研基金资助项目(201012MS010)

摘　　要：基于词频反文档频率(term frequency inverse document frequency,TFIDF)的现有文本特征提取算法及其改进算法未能考虑类别内部词语之间的语义关联,如果脱离语义,提取出的特征不能很好地刻画文档的内容。为准确提取特征,在信息熵与信息增益的基础上,加入词语的语义关联因素,实现融合语义信息的特征提取,进而提出语义和信息增益相结合的TFIDF改进算法,该算法弥补了统计方法丢失语义信息的弊端。实验结果表明,该算法有效地提高了文本分类的精准率。Both the traditional and improved term frequency-inverse document frequency（TFIDF） algorithms ignored the difference of distributions among different categories in feature extraction.Due to the lacking of consideration of semantic relationships within some certain categories,the selected feature word cannot describe the contents of the document correctly and accurately.In order to select feature more accurately,in this paper,based on the previous improvements,introduced the semantic association of words to analyze the semantic of text,redesigned the weights equation,and proposed the new TFIDF algorithm combined with semantic and information gain.The developed algorithm can make up for the shortcomings of the lack of semantic information in statistical method.Experimental results illustrate that the improved algorithm can effectively improve text classification accuracy.

关 键 词：词频反文档频率 特征提取 语义关联 信息增益 文本分类 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]