结合改进的CHI统计方法的TF-IDF算法优化  

Optimization of TF-IDF algorithm combined with improved CHI statistical method

作　　者：马莹 赵辉[1] 李万龙[1] 庞海龙[1] 崔岩 Ma Ying;Zhao Hui;Li Wanlong;Pang Hailong;Cui Yan(College of Computer Science & Engineering,Changchun University of Technology,Changchun 130012,China)

机构地区：[1]长春工业大学计算机科学与工程学院

出　　处：《计算机应用研究》2019年第9期2596-2598,2603,共4页Application Research of Computers

基　　金：国家自然科学基金资助项目(61472049);吉林省教育厅“十二五”科学技术研究项目(2014132)

摘　　要：为了克服传统的CHI统计方法存在特征项出现频率与类别负相关的情况和某一个特征项存在于某一个文本中的概率问题,针对传统的CHI统计方法引入了负相关判定、频度等重要因素进行了改进,并结合语义相似度的计算方法对TF-IDF算法进行了优化,在WEKA软件上采用了KNN(K-nearest neighbor)分类器和支持向量机(SVM)分类器分别对微博情感语料进行分类,该实验结果表明,新方法在文本分类的准确性上有明显的提高。In order to overcome the traditional CHI statistical method, there was a negative correlation between the frequency of feature items and the category, and a probability problem that a feature item existed in a text, The traditional CHI statistical method was improved by introducing some important factors such as negative correlation judgment and frequency, and the TF-IDF algorithm was optimized by combining the calculation method of semantic similarity. The K- nearest neighbor (KNN) classifier and support vector machine (SVM) classifier were respectively used in WEKA software to classify the Weibo emotional corpus the experimental results show that the new method has obvious improvement on the accuracy of text classification.

关 键 词：文本分类 CHI统计 TF-IDF算法 特征选择 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]