随机初始化神经网络剪枝的稀疏二值规划方法  

Sparse Binary Programming Method for Pruning of Randomly Initialized Neural Networks

作　　者：陆林[1,2] 季繁繁 袁晓彤 LU Lin;JI Fanfan;YUAN Xiaotong(School of Automation,Nanjing University of Information Science and Technology,Nanjing 210044,China;Jiangsu Key Laboratory of Big Data Analysis Technology,Nanjing 210044,China;School of Computer,Nanjing University of Information Science and Technology,Nanjing 210044,China)

机构地区：[1]南京信息工程大学自动化学院,南京210044 [2]江苏省大数据分析技术重点实验室,南京210044 [3]南京信息工程大学计算机学院,南京210044

出　　处：《计算机工程与应用》2023年第8期138-147,共10页Computer Engineering and Applications

基　　金：科技创新2030-“新一代人工智能”重大项目(2018AAAA0100400);国家自然科学基金(61876090,61936005,U21B2049)。

摘　　要：传统深度神经网络剪枝方法往往以预训练模型为初始网络并需要在剪枝后进行微调。受到近年来edgepopup等基于随机初始化网络的剪枝算法优异性能的启发,提出了一种基于稀疏二值规划的随机初始化网络剪枝算法。该算法将剪枝训练过程建模为一个稀疏二值约束优化问题。其核心思想是利用稀疏二值规划来学习一个二值掩膜,利用该掩膜可以从随机初始化的神经网络上裁剪出一个未经训练却性能良好的稀疏网络。与之前基于随机初始化网络的剪枝算法相比,该算法找到的稀疏网络在多个稀疏度下都具有更好的分类泛化性能。与edge-popup算法相比,在ImageNet数据集分类任务中,模型在稀疏度为70%时精度提升7.98个百分点。在CIFAR-10数据集分类任务中,模型在稀疏度为50%时精度提升2.48个百分点。Classical pruning algorithms for deep neural networks typically need to pre-train the model before pruning and fine tune the sparse network after pruning.Inspired by the recent remarkable success of random initialization based pruning methods such as edge-popup,this paper proposes a sparse binary programming method for random network pruning.The algorithm models the pruning training process as a sparse binary constrained optimization problem.The core idea is to use sparse binary programming to learn a binary mask,using which an untrained but well-performing sparse network can be pruned from a randomly initialized neural network.Compared with previous pruning algorithms based on randomly initialized networks,the sparse network found by this algorithm has better classification generalization performance at multiple sparsity degrees.Compared with the edge-popup algorithm,the model improves the accuracy by 7.98 percentage points at 70%sparsity in the ImageNet dataset classification task.In the CIFAR-10 dataset classification task,the model improves 2.48 percentage points accuracy at 50%sparsity.

关 键 词：神经网络剪枝 随机初始化 二值掩膜 二值规划 稀疏优化 

分 类 号：TP39[自动化与计算机技术—计算机应用技术]