融合动态掩码注意力与多教师多特征知识蒸馏的文本分类  

Text Classification by Combining Dynamic Mask Attention and Multi-teacher Multi-feature Knowledge Distillation

作　　者：王润周 张新生[1] 王明虎[1] WANG Runzhou;ZHANG Xinsheng;WANG Minghu(School of Management,Xi'an University of Architecture and Technology,Xi'an,Shaanxi 710055,China)

机构地区：[1]西安建筑科技大学管理学院,陕西西安710055

出　　处：《中文信息学报》2024年第3期113-129,共17页Journal of Chinese Information Processing

基　　金：陕西省重点产业创新链(群)-工业领域项目(2022ZDLGY06-04);陕西省社科界重大理论与现实问题研究联合项目(2022HZ1522)。

摘　　要：知识蒸馏技术可以将大规模模型中的知识压缩到轻量化的模型中,在文本分类任务中实现更高效的推断。现有的知识蒸馏方法较少同时考虑多种教师与多个特征层之间的信息融合。此外,蒸馏过程采用全局填充,未能动态关注数据中的有效信息。为此,该文提出一种融合动态掩码注意力机制与多教师多特征知识蒸馏的文本分类模型,不仅引入多种教师模型(RoBERTa、Electra)的知识源,还兼顾不同教师模型在多个特征层的语义信息,并通过设置动态掩码模型注意力机制使得蒸馏过程动态关注不等长数据,减少无用填充信息的干扰。在4种公开数据集上的实验结果表明,经过蒸馏后的学生模型(TinyBRET)在预测性能上均优于其他基准蒸馏策略,并在采用教师模型1/10的参数量、约1/2的平均运行时间的条件下,取得与两种教师模型相当的分类结果,平均准确率仅下降4.18%和3.33%,平均F 1值仅下降2.30%和2.38%。其注意力热度图也表明动态掩码注意力机制切实加强关注了数据尾部与上下文信息。The knowledge distillation technique compresses knowledge from large-scale models into lightweight models,improving the efficiency of text classification.This paper introduces a text classification model that combines a dynamic mask attention mechanism and multi-teacher,multi-feature knowledge distillation.It leverages knowledge sources from various teacher models,including Roberta and Electra,while considering semantic information across different feature layers.The dynamic mask attention mechanism adapts to varying data lengths,reducing interference from irrelevant padding.Experimental results on four publicly available datasets demonstrate that the student model(TinyBERT)distilled by the proposed method outperforms other benchmark distillation strategies.Remarkably,with only 1/10 of the teacher model's parameters and approximately half the average runtime,it achieves classification results comparable to the two teacher models,with only a marginal decrease in accuracy(4.18%and 3.33%)and F 1 value(2.30%and 2.38%).The attention heat map indicates that the dynamic mask attention mechanism enhances focus on the effective information of the data.

关 键 词：动态掩码注意力 多教师多特征 知识蒸馏 文本分类 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]