基于多层知识互学习的在线对抗蒸馏  

Online Adversarial Distillation Based on Multi-layer Knowledge Mutual Learning

作　　者：李舒意 扈红超 杨晓晗 程国振[1,2] 郭威 LI Shuyi;HU Hongchao;YANG Xiaohan;CHENG Guozhen;GUO Wei(Information Engineering University,Zhengzhou 450001,China;Key Laboratory of Cyberspace Security,Ministry of Education,Zhengzhou 450001,China)

机构地区：[1]信息工程大学,河南郑州450001 [2]网络空间安全教育部重点实验室,河南郑州450001

出　　处：《信息工程大学学报》2025年第2期209-216,共8页Journal of Information Engineering University

基　　金：河南省重大科技专项(221100211200-02)。

摘　　要：针对对抗蒸馏中静态教师知识的可靠性逐渐下降和知识单一导致的鲁棒精度提升有限的问题,提出一种基于多层知识互学习的在线对抗蒸馏方法。首先加权融合一组对抗训练中的学生的模型参数构建参数融合模型(PFM),最小化PFM的对抗损失以帮助学生寻找更平坦的损失最小值。其次,鼓励学生在对抗样本的预测分布上的类间排名关系模仿所有模型的平均原始预测,以确保输出层可靠的软标签知识的互学习。最后,最大化学生之间基于特征通道级别的样本间关系的相似性,以利用中间层更丰富的特征结构化知识补充单一的软标签知识中的有限信息。实验结果表明,相较于互对抗训练(MAT),所提方法训练的ResNet-18网络在防御由投影梯度下降法(PGD)生成的对抗样本上的测试准确率分别提升2.05和2.19个百分点。An online adversarial distillation method,based on multi-layer knowledge mutual learning,is proposed to address the issues of limited improvement in robust accuracy,which are caused by the declining reliability of static teacher knowledge and the underutilization of knowledge in existing adversarial distillation methods.A parameter fusion model(PFM)is constructed by weightedly fusing the model parameters of a group of students,with the adversarial loss of PFM being minimized to assist students in finding flatter loss minima.To ensure mutual learning of reliable soft label knowledge at the output layer,students are encouraged to mimic the average clean predictions of all models,in terms of inter-class ranking relationships within the predictive distribution of adversarial examples.Subsequently,to utilize the richer structured knowledge of the middle layer for supplementing the limited information in soft labels,the similarity of sample relationships at the feature channel level among students is maximized.Experimental results demonstrate that,compared to mutual adversarial training(MAT)ResNet-18 network trained by the method exhibits an improvement in test accuracy against adversarial examples generated by projected gradient descent(PGD)by 2.05 and 2.19 percentage point,respectively.

关 键 词：参数融合 软标签知识 关系知识 在线对抗蒸馏 对抗训练 对抗样本 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术] TP183[自动化与计算机技术—计算机科学与技术]