Two-Stage Approach for Targeted Knowledge Transfer in Self-Knowledge Distillation  

作　　者：Zimo Yin Jian Pu Yijie Zhou Xiangyang Xue 

机构地区：[1]the School of Computer Science,Fudan University [2]the Institute of Science and Technology for Brain Inspired Intelligence,Fudan University [3]IEEE

出　　处：《IEEE/CAA Journal of Automatica Sinica》2024年第11期2270-2283,共14页自动化学报（英文版）

基　　金：supported by the National Natural Science Foundation of China (62176061)。

摘　　要：Knowledge distillation(KD) enhances student network generalization by transferring dark knowledge from a complex teacher network. To optimize computational expenditure and memory utilization, self-knowledge distillation(SKD) extracts dark knowledge from the model itself rather than an external teacher network. However, previous SKD methods performed distillation indiscriminately on full datasets, overlooking the analysis of representative samples. In this work, we present a novel two-stage approach to providing targeted knowledge on specific samples, named two-stage approach self-knowledge distillation(TOAST). We first soften the hard targets using class medoids generated based on logit vectors per class. Then, we iteratively distill the under-trained data with past predictions of half the batch size. The two-stage knowledge is linearly combined, efficiently enhancing model performance. Extensive experiments conducted on five backbone architectures show our method is model-agnostic and achieves the best generalization performance.Besides, TOAST is strongly compatible with existing augmentation-based regularization methods. Our method also obtains a speedup of up to 2.95x compared with a recent state-of-the-art method.

关 键 词：Cluster-based regularization iterative prediction refinement model-agnostic framework self-knowledge distillation(SKD) two-stage knowledge transfer 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]