前缀标签嵌入的参数高效微调与预测分数引导Mixup的文本分类方法  

Efficient parameter fine-tuning of prefix label embedding and prediction score-guided Mixup text classification method

作　　者：庞宇宏 线岩团[1,2] 相艳[1,2] 黄于欣 PANG Yuhong;XIAN Yantuan;XIANG Yan;HUANG Yuxin(Faculty of Information Engineering and Automation,Kunming University of Science and Technology,Kunming 650500,China;Yunnan Key Laboratory of Artificial Intelligence,Kunming University of Science and Technology,Kunming 650500,China)

机构地区：[1]昆明理工大学信息工程与自动化学院,云南昆明650500 [2]昆明理工大学云南省人工智能重点实验室,云南昆明650500

出　　处：《微电子学与计算机》2025年第2期31-38,共8页Microelectronics & Computer

基　　金：国家自然科学基金项目(62266028);云南重大科技专项计划(202202AD080003)。

摘　　要：文本分类旨在根据文本的特征,将其划分到一个或多个类别中。目前,在面对文本对抗攻击时,传统的深度学习模型和微调预训练语言模型往往面临着过拟合问题。由于训练数据的特定性,使得模型在处理与训练数据分布不同的对抗性样本时,无法充分泛化,进而降低了模型在对抗性攻击场景中的鲁棒性。一些参数高效的微调方法采用轻量级的模型结构,由于相对较低的表达能力使得模型无法有效捕捉对抗性攻击的复杂特征,导致其鲁棒性差。此外,在模型分类过程中,无论是用于分类的特征向量还是起到引导作用的前缀向量,对分类结果的影响机制尚未得到清晰的认识导致模型的可解释性差。本文提出一个新的方法,将前缀标签嵌入与预训练语言模型融合,在分类层面引入标签与文本相似度的打分机制,通过预测分数引导下的Mixup,有效地挖掘与分类密切相关的特征,缓解过拟合问题,提升模型的鲁棒性。同时结合多头机制,使模型获得更加丰富的特征表达,提升模型可解释性。实验表明,该框架在保持参数高效微调前提下提高了针对4种不同类型的文本攻击的鲁棒性,同时保持了干净文本的可比准确性。Text classification aims to classify text into one or more categories based on its characteristics.Currently,when facing text adversarial attacks,traditional deep learning models and fine-tuned pre-trained language models often face overfitting problems.Due to the specificity of the training data,the model cannot fully generalize when processing adversarial samples with different distributions from the training data,thereby reducing the robustness of the model in adversarial attack scenarios.Similarly,some parameter-efficient fine-tuning methods use lightweight model structures.Due to the relatively low expression ability,the model cannot effectively capture the complex characteristics of adversarial attacks,resulting in poor robustness.In addition,in the model classification process,whether it is the feature vector used for classification or the prefix vector that plays a guiding role,their impact mechanism on the classification results has not been clearly understood,resulting in poor interpretability of the model.Therefore,this paper proposes a new method that integrates prefix label embedding and pre-trained language models,and introduces a scoring mechanism for label and text similarity at the classification level.Through Mixup guided by the prediction score,it effectively mines information closely related to classification.features,alleviate the over-fitting problem and improve the robustness of the model.At the same time,it is combined with the multi-head mechanism to enable the model to obtain richer feature expressions and improve the interpretability of the model.Experiments show that this framework greatly improves the robustness against four different types of text attacks while maintaining efficient fine-tuning of parameters,while maintaining comparable accuracy on clean text.

关 键 词：文本分类 鲁棒性 预训练语言模型 前缀标签嵌入 Mixup数据增强 参数高效微调 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]