基于多教师多学生知识蒸馏的情感分类方法  

Multiple-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification

作　　者：常晓琴 李雅梦 李子成 李寿山[1] CHANG Xiaoqin;LI Yameng;LI Zicheng;LI Shoushan(School of Computer Science and Technology,Soochow University,Suzhou,Jiangsu 215006,China)

机构地区：[1]苏州大学计算机科学与技术学院,江苏苏州215006

出　　处：《中文信息学报》2024年第10期127-134,共8页Journal of Chinese Information Processing

基　　金：国家自然科学基金(62076176)。

摘　　要：预训练语言模型在情感分类任务中取得了卓越的性能提升。然而,预训练模型的巨额参数量和缓慢的推理速度成为这些模型应用落地的主要障碍。知识蒸馏是一种将知识从大型的预训练教师模型转移到小型学生模型的技术。不同于现有的单教师或单学生蒸馏模型,该文提出一种基于多教师和多学生蒸馏的集成蒸馏方法。该方法既可以充分利用不同教师模型的不同知识,又可以弥补单个学生学习能力不足的缺点。此外,该文使用了大量情感分类任务相关的未标注样本来提高蒸馏性能。实验结果表明,该文提出的方法在情感分类任务上能够在基本保持教师模型的分类性能的基础上,减少97.8%~99.5%参数量,并提升了176~645倍的CPU推理速度。Pre-trained language models(PLMs)have achieved great success in sentiment classification.To compress the huge model parameters and speed up the inference,the knowledge distillation is a popular solution to transfer knowledge from a large pre-trained teacher model to a compacted student model.In contrast to the existing single-teacher or single-student models,this paper proposes a multiple-teacher and multiple-student knowledge distillation approach to exploit the different knowledge of teacher models and improve the learning ability of a single student.We also leverage large-scale unlabeled data to improve the performance of students.Experiments on sentiment classification show that,compared with the teacher model,our approach achieves similar performance with 97.8%~99.5%less parameters and 176~645 times CPU inference speed.

关 键 词：情感分类 知识蒸馏 集成学习 模型压缩 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]