An Improved Knowledge Distillation Algorithm and Its Application to Object Detection  

作　　者：Min Yao Guofeng Liu Yaozu Zhang Guangjie Hu 

机构地区：[1]School of Information Engineering,Shanghai Maritime University,Shanghai,201306,China [2]Baidu,Beijing,100000,China [3]Shanghai Freesense Technology Co.,Ltd.,Shanghai,200000,China

出　　处：《Computers, Materials & Continua》2025年第5期2189-2205,共17页计算机、材料和连续体(英文)

基　　金：funded by National Natural Science Foundation of China(61603245).

摘　　要：Knowledge distillation(KD)is an emerging model compression technique for learning compact object detector models.Previous KD often focused solely on distilling from the logits layer or the feature intermediate layers,which may limit the comprehensive learning of the student network.Additionally,the imbalance between the foreground and background also affects the performance of the model.To address these issues,this paper employs feature-based distillation to enhance the detection performance of the bounding box localization part,and logit-based distillation to improve the detection performance of the category prediction part.Specifically,for the intermediate layer feature distillation,we introduce feature resampling to reduce the risk of the student model merely imitating the teacher model.At the same time,we incorporate a Spatial Attention Mechanism(SAM)to highlight the foreground features learned by the student model.In terms of output layer feature distillation,we divide the traditional distillation targets into target-class objects and non-target-class objects,aiming to improve overall distillation performance.Furthermore,we introduce a one-to-many matching distillation strategy based on Feature Alignment Module(FAM),which further enhances the studentmodel’s feature representation ability,making its feature distribution closer to that of the teacher model,and thus demonstrating superior localization and classification capabilities in object detection tasks.Experimental results demonstrate that our proposedmethodology outperforms conventional distillation techniques in terms of object detecting performance.

关 键 词：Deep learning model compression knowledge distillation object detection 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]