面向知识蒸馏的多助教动态设置方法  

Multi-assistant Dynamic Setting Method for Knowledge Distillation

作　　者：司悦航 成清[1,2] 黄金才 SI Yuehang;CHENG Qing;HUANG Jincai(Laboratory for Big Data and Decision,National University of Defense Technology,Changsha 410073,China;Hunan Advanced Technology Research Institute,Changsha 410072,China)

机构地区：[1]国防科技大学大数据与决策实验室,长沙410073 [2]湖南先进技术研究院,长沙410072

出　　处：《计算机科学》2025年第5期241-247,共7页Computer Science

摘　　要：知识蒸馏在目标识别的模型压缩等关键领域受到重视。通过深入研究知识蒸馏的效率并分析教师模型和学生模型间知识传递的特点,发现合理设置助教模型可以显著缩小教师和学生之间的性能差距。然而,助教模型的规模和数量的不合理选择会对学生产生负面影响。因此,提出了一种创新的多助教知识蒸馏训练框架,通过动态调整助教的数量和规模,以优化知识从教师向学生传递的过程,从而提高学生模型的训练准确率。此外,还设计了一种动态停止知识蒸馏的策略,设置不同训练方法的学生模型作为对照组,实现对知识蒸馏停止回合的个性化设计,进一步提升学生模型的训练效率,并构建更精简高效的多助教知识蒸馏框架。通过在公开数据集上进行实验,证明了提出的面向知识蒸馏的多助教动态设置方法的有效性。Knowledge distillation is increasingly gaining attention in key areas such as model compression for object recognition.Through in-depth research into the efficiency of knowledge distillation and an analysis of the characteristics of knowledge transfer between the teacher and student models,it is found that the reasonable setting of an assistant model can significantly reduce the performance gap between the teacher and student.However,the unreasonable choice of the scale and number of assistant models can have a negative impact on the student.Therefore,this paper proposes an innovative multi-assistant knowledge distillation training framework,which optimizes the process of knowledge transfer from the teacher to the student by dynamically adjusting the number and scale of assistant models,thereby improving the training accuracy of the student model.In addition,this paper also designs a dynamic stopping strategy for knowledge distillation,sets student models with different training methods as a control group,and achieves personalized design of the stopping rounds for knowledge distillation,further improving the training efficiency of the student model and constructing a more streamlined and efficient multi-assistant knowledge distillation framework.Experiments on public datasets prove the effectiveness of the proposed multi-assistant dynamic setting method for knowledge distillation.

关 键 词：知识蒸馏 目标识别 多助教 动态设置 DSKD 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]