基于信息熵与词长信息改进的TFIDF算法  

Improved TFIDF algorithm based on information entropy and word length information

作　　者：金燕[1] 黄杰 JIN Yan;HUANG Jie(College of Information Engineering,Zhejiang University of Technology,Hangzhou 310023,China)

机构地区：[1]浙江工业大学信息工程学院,浙江杭州310023

出　　处：《浙江工业大学学报》2021年第2期203-209,共7页Journal of Zhejiang University of Technology

基　　金：浙江省自然科学基金资助项目(LY17F010015)。

摘　　要：针对文本分类中传统的TFIDF特征提取算法的缺陷,引入信息熵与词长信息改进TFIDF算法。传统的TFIDF算法中忽略了词长信息,词长不同能够表达的信息也不同,同时还忽略了文本中特征词的分布特征。改进的TFIDF算法中加入了表达词长信息的因子并且引入词条信息熵来反映特征词在文本中的分布特征,实验比较了其与TFIDF、TFIDFL等算法在相同数据集上使用逻辑回归分类器的分类准确率。改进的算法平均准确率比TFIDF算法高了7.34%,比TFIDFL算法高了5.99%,结果表明引入信息熵与词长信息改进TFIDF算法能够有效提升分类准确率。Aiming at the defects of traditional TFIDF feature extraction algorithm in text classification,information entropy and word length information are introduced to improve the TFIDF algorithm.In the traditional TFIDF algorithm,word length information is ignored,and different word length can express different information.Meanwhile,it ignores the distribution characteristics of feature words in the text.A factor expressing word length information is added to the improved TFIDF algorithm and the entry information entropy is used to reflect the feature word distribution in the text.The experiment compares its classification accuracy with TFIDF,TFIDFL and other algorithms using logistic regression classifier on the same data set.The average accuracy of the improved algorithm is 7.34%higher than that of the TFIDF algorithm,and 5.99%higher than that of the TFIDFL algorithm.The results show that the introduction of information entropy and word length information to improve the TFIDF algorithm can effectively improve the classification accuracy.

关 键 词：TFIDF 信息熵 特征提取 文本分类 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]