基于改进DINO的联合蒸馏图像分类  

Co-Distillation Image Classification Based on Improved DINO

作　　者：尹威 林贵敏 YIN Wei;LIN Guimin(College of Physics and Electronic Information Engineering,Minjiang University,Fuzhou,Fujian 350108,China;College of Photonic and Electronic Engineering,Fujian Normal University,Fuzhou,Fujian 350117,China)

机构地区：[1]闽江学院物理与电子信息工程学院,福建福州350108 [2]福建师范大学光电与信息工程学院,福建福州350117

出　　处：《闽江学院学报》2024年第5期42-50,共9页Journal of Minjiang University

基　　金：福建省自然科学基金项目(2019J01761,2023J011401);福建省科技计划项目(2020J02024);福建省本科高校教育教学研究项目(重大项目)(FBJY20230095)。

摘　　要：DINO(self-distillation with no label)首次将自监督学习与transformer结合。为结合卷积网络的局部性优势,提出一个三分支的网络模型DINO+,即在DINO的基础上添加一个卷积蒸馏模块,给DINO中的transformer进行知识蒸馏,从而将卷积网络与transformer结合。蒸馏后的ViT(vision transformer)在STL-10、CIFAR-10数据集上的分类准确率分别上升了5.7%、4.8%,且优于其他自监督模型,证明了提出方法的有效性。DINO(self-distillation with no label)combines self-supervised learning with transformer for the first time.In order to combine the locality advantage of convolutional network,a three-branch network model named DINO+is proposed,which adds a convolution distillation module to DINO to distill knowledge to transformer in DINO,thus combining the convolutional network with transformer.The classification accuracy of the distilled ViT(vision transformer)on STL-10 and CIFAR-10 increases by 5.7%and 4.8%respectively,and is better than other self-supervised models,demonstrating the effectiveness of the proposed method.

关 键 词：DINO 自监督学习 卷积蒸馏模块 知识蒸馏 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]