一种基于权重预处理的中文文本分类算法  

A Chinese Text Classification Algorithm Based on Weight Preprocessing

作　　者：何铠 管有庆[1] 龚锐 HE Kai;GUAN You-qing;GONG Rui(School of Internet of Things,Nanjing University of Posts and Telecommunications,Nanjing 210003,China)

机构地区：[1]南京邮电大学物联网学院,江苏南京210003

出　　处：《计算机技术与发展》2022年第3期40-45,53,共7页Computer Technology and Development

基　　金：江苏省高校自然科学研究计划项目(05KJD520146)。

摘　　要：文本分类是NLP(natural language processing,自然语言处理)处理技术的重要分支。信息检索、文本挖掘作为自然语言处理领域的关键技术,给人们的生活带来了许多便利,而文本分类正是这些关键技术开展的重要基础。文本分类作为自然语言处理研究的一个热点,其主要原理是将文本数据按照一定的分类规则实现自动化分类。目前常见的文本分类方式主要分为基于机器学习和基于深度学习两种,它们的本质是通过计算机自主学习从而提取文本信息中的规则来进行分类。针对数据量较小、硬件运算能力较低的应用场景,往往使用基于机器学习算法而衍生的文本分类模型。该文以期刊论文作为实验数据,研究中文文本分类问题,在改进传统词频算法的基础上提出了一种基于权重预处理的中文文本分类算法PRE-TF-IDF(pre-processing term frequency inverse document frequency)。传统词频算法在对词加权时仅考虑词的出现频率而不考虑词在文本中的位置;PRE-TF-IDF算法在TF-IDF(term frequency inverse document frequency)算法的基础上增加权重预处理和词密度权重两个环节。实验结果显示PRE-TF-IDF算法能够有效提高文本分类的准确性。Text classification is an important branch of NLP(natural language processing).Information retrieval and text mining,as key technologies in the field of natural language processing,have brought a lot of convenience to people’s lives,and text classification is an important basis for the development of those key technologies.Text classification is a hot topic in natural language processing.The main principle of text classification is to automatically classify text data according to certain classification rules.At present,common text classification methods are mainly divided into two types:machine learning and deep learning.Their essence is to extract rules from text information through computer autonomous learning for classification.The text classification model derived from a machine learning algorithm is often used for application scenarios with a small amount of data and low hardware computing power.We take journal papers as experimental data to study the classification of Chinese text.Based on improving the traditional word frequency algorithm,a Chinese text classification algorithm based on weight preprocessing,PRE-TF-IDF(pre-processing term frequency inverse document frequency),is proposed.The traditional word frequency algorithm only considers the occurrence frequency of words but does not consider the position of words in the text when weighing words.Based on the TF-IDF(term frequency inverse document frequency)algorithm,the PRE-TF-IDF algorithm has two additional steps:weight preprocessing and word density weight.Experiment shows that the PRE-TF-IDF algorithm can effectively improve the accuracy of text classification.

关 键 词：自然语言处理 词频算法 中文文本分类 权重预处理 词密度权重 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]