基于金字塔池化以及掩码生成的特征知识蒸馏  

Feature Knowledge Distillation Based on Pyramid Pooling and Mask Generation

作　　者：陆新才 孙占全 王贺 李庆蓬 Xincai Lu;Zhanquan Sun;He Wang;Qingpeng Li(School of Optical-Electrical and Computer Engineering,University of Shanghai for Science and Technology,Shanghai;School of Economics,Henan University,Zhengzhou Henan)

机构地区：[1]上海理工大学光电信息与计算机工程学院,上海 [2]河南大学经济学院,河南郑州

出　　处：《建模与仿真》2025年第2期279-290,共12页Modeling and Simulation

摘　　要：知识蒸馏(KD)的目标是将知识从大型教师网络传递到轻量级的学生网络中去。主流的KD方法可以被分为Logit蒸馏和特征蒸馏。基于特征的知识蒸馏是KD的重要组成部分,它利用中间层来监督学生网络的训练过程。然而,中间层的潜在不匹配可能会在训练过程中适得其反,并且目前的学生模型往往直接通过模仿老师的特征来学习。针对这一问题,本文提出了一种新的知识蒸馏框架,称为解耦空间金字塔池知识蒸馏,以区分特征图中区域的重要性。同时,本文还提出了一种掩码生成特征蒸馏模块,指导学生模型通过一个块生成而不是模仿教师的完整特征。与之前复杂的蒸馏方法相比,本文提出的方法在CIFAR-100和Tiny-ImageNet数据集上取得了更高的知识蒸馏模型分类结果。The goal of Knowledge Distillation(KD)is to transfer knowledge from a large teacher network to a lightweight student network.Mainstream KD methods can be divided into logit distillation and feature distillation.Feature-based knowledge distillation is a critical component of KD,utilizing intermediate layers to supervise the training process of the student network.However,potential mismatches in intermediate layers may backfire during training,and current student models often learn directly by imitating the teacher’s features.To address this issue,this paper proposes a novel distillation framework called Decoupled Spatial Pyramid Pooling Knowledge Distillation,which distinguishes the importance of regions in feature maps.This paper also introduces a mask-based feature distillation module,which guides the student model to generate features from a block rather than mimicking the complete features of the teacher model.Compared to previous complex distillation methods,the proposed approach achieves superior classification results on the CIFAR-100 and Tiny-ImageNet datasets.

关 键 词：模型压缩 知识蒸馏 特征蒸馏 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]