基于特征重建的知识蒸馏方法  

Knowledge Distillation Based on Feature Reconstruction

作　　者：郭俊伦 彭书华[1] 李俊杰 GUO Jun-lun;PENG Shu-hua;LI Jun-jie(School of Automation,Beijing Information Science and Technology University,Beijing 100192)

机构地区：[1]北京信息科技大学自动化学院,北京100192

出　　处：《现代计算机》2020年第29期43-47,73,共6页Modern Computer

基　　金：国家自然科学基金(No.61801032)。

摘　　要：为了解决深度神经网络压缩后精度损耗大的问题,提出一种基于知识蒸馏的模型压缩架构。该架构的创新点在于结合了图片风格迁移实验中的特征重建方法,在训练学生网络时,将学生网络的浅层特征分组嵌入已经训练好的教师网络中对深度特征进行特征重建,从而达到学生网络的输出结果逼近教师网络的目的。实验结果表明,在图像分类任务中,网络宽度不变的情况下,将模型压缩为原模型的一半,在Cifar10数据集下准确率仅损失约1.5%;在Cifar100数据集下准确率仅损失约3%。由此可知,此架构可应用于图像分类任务的模型压缩中。In order to solve the problem of large precision loss after deep neural network compression,a model compression architecture based on knowledge distillation is proposed.The innovation of this architecture is to combine the feature reconstruction method in the picture style transfer experiment.when training the student network,the student network’s Shallow feature grouping is embedded in the trained teach⁃er network to perform feature reconstruction on the deep features,so as to achieve the goal that the output of the student network is close to the teacher network.The experimental results show that in the task of image classification,when the network width is constant,the model is compressed to half of the original model,and the accuracy loss is only about 1.5%in the Cifar10,and only about 3%in the Cifar100.Therefore,this architecture can be applied to model compression of image classification tasks.

关 键 词：深度神经网络 模型压缩 知识蒸馏 特征重建 风格迁移 图像分类 

分 类 号：TP183[自动化与计算机技术—控制理论与控制工程] TP391.41[自动化与计算机技术—控制科学与工程]