文本分类中一种改进的特征项权重计算方法  

An Improved Feature Weighting Method in Text Classification

作　　者：张琳 李朝辉 ZHANG Lin;LI Zhaohui(School of Maritime Economics and Management,Dalian Maritime University,Dalian 116026,China)

机构地区：[1]大连海事大学航运经济与管理学院,辽宁大连116026

出　　处：《福建师范大学学报（自然科学版）》2020年第2期49-54,共6页Journal of Fujian Normal University：Natural Science Edition

基　　金：辽宁省社会科学规划基金项目(L18CTQ004);中国博士后科学基金资助项目(2015M571292);中央高校基本科研业务费专项资金资助项目(3132019234、3132019228);大连海事大学教学改革项目(2018Y48)。

摘　　要：TF-IDF方法是文本向量化过程中一种常用的特征项权重计算方法,衡量的是特征项在整个文档集中的重要性.针对文本分类过程中TF-IDF方法未能体现特征项对类别的区分能力和对类别的代表性问题,基于文档类别,结合特征项的类间区分度和类内贡献度,提出一种改进的TF-IDF权重计算方法,并采用KNN和SVM模型对改进后算法的分类性能进行了验证.实验结果表明,与传统的TF-IDF方法相比,改进后的权重计算方法不仅在整个测试数据集上能够取得较高的宏平均精确率、宏平均召回率和宏平均F1,而且使测试数据集绝大部分类别的分类性能得到了较大提升.因此,改进后的TF-IDF权重计算方法是有效且可行的.TF-IDF is a commonly used method to weigh features in the process of text vectorization,which measures the importance of features in the whole data set.Aiming at the problem that TF-IDF method fails to reflect the ability of features to distinguish and represent classes in text classification,this paper proposed an improved TF-IDF method based on document category,and the inter-class discrimination and intra-class contribution of features.Besides,this paper used KNN and SVM models to validate the performance of the improved method in text classification.Result shows that compared with the traditional TF-IDF method,the improved one not only achieved higher macro-average precision,macro-average recall and macro-average F1 on the whole test dataset,but also greatly improved the classification performance on most categories of the test dataset.Therefore,the improved method is effective and feasible.

关 键 词：TF-IDF 类间区分度 类内贡献度 文本分类 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]