Swin Transformer轻量化:融合权重共享、蒸馏与剪枝的高效策略  

Swin Transformer lightweight:an efficient strategy that combines weight sharing,distillation and pruning

作　　者：韩博 周顺 范建华 魏祥麟 胡永杨 朱艳萍[1] HAN Bo;ZHOU Shun;FAN Jianhua;WEI Xianglin;HU Yongyang;ZHU Yanping(School of Electronic and Information Engineering,Nanjing University of Information Science and Technology,Nanjing 210044,China;The Sixty-third Research Institute,National University of Defense Technology,Nanjing 210007,China)

机构地区：[1]南京信息工程大学电子与信息工程学院,江苏南京210044 [2]国防科技大学第六十三研究所,江苏南京210007

出　　处：《电信科学》2024年第9期66-74,共9页Telecommunications Science

摘　　要：偏移窗口的分层视觉转换器(Swin Transformer)因其优秀的模型能力而在计算机视觉领域引起了广泛的关注,然而Swin Transformer模型有着较高的计算复杂度,限制了其在计算资源有限设备上的适用性。为缓解该问题,提出一种融合权重共享及蒸馏的模型剪枝压缩方法。首先,在各层之间实现了权重共享,并添加变换层实现权重变换以增加多样性。接下来,构建并分析变换块的参数依赖映射图,构建分组矩阵F记录所有参数之间的依赖关系,确定需要同时剪枝的参数。最后,蒸馏被用于恢复模型性能。在ImageNet-Tiny-200公开数据集上的试验表明,在模型计算复杂度减少32%的情况下,最低仅造成约3%的性能下降,有效降低了模型的计算复杂度。为实现在计算资源受限环境中部署高性能人工智能模型提供了一种解决方案。Swin Transformer,as a layered visual transformer with shifted windows,has attracted extensive attention in the field of computer vision due to its exceptional modeling capabilities.However,its high computational complexity limits its applicability on devices with constrained computational resources.To address this issue,a pruning compression method was proposed,integrating weight sharing and distillation.Initially,weight sharing was implemented across layers,and transformation layers were added to introduce weight transformation,thereby enhancing diversity.Subsequently,a parameter dependency mapping graph for the transformation blocks was constructed and analyzed,and a grouping matrix F was built to record the dependency relationships among all parameters and identify param-eters for simultaneous pruning.Finally,distillation was then employed to restore the model’s performance.Experiments conducted on the ImageNet-Tiny-200 public dataset demonstrate that,with a reduction of 32%in model computational complexity,the proposed method only results in approximately a 3%performance degradation at minimum.It provides a solution for deploying high-performance artificial intelligence models in environments with limited computational resources.

关 键 词：偏移窗口的分层视觉转换器 模型轻量化 推理加速 剪枝 蒸馏 权重共享 

分 类 号：TP183[自动化与计算机技术—控制理论与控制工程]