深度神经网络的自适应联合压缩方法  

Adaptive joint compression method for deep neural networks

作　　者：姚博文 彭喜元[1] 于希明 刘连胜[1] 彭宇[1] Yao Bowen;Peng Xiyuan;Yu Ximing;Liu Liansheng;Peng Yu(Department of Test and Control Engineering,Harbin Institute of Technology,Harbin 150080,China)

机构地区：[1]哈尔滨工业大学测控工程系,哈尔滨150080

出　　处：《仪器仪表学报》2023年第5期21-32,共12页Chinese Journal of Scientific Instrument

摘　　要：现有模式单一且固定的深度神经网络压缩方法受限于精度损失,而难以对模型进行充分压缩,致使压缩后模型在实际部署时仍需消耗大量成本高昂且容量有限的存储资源,对其在边缘端的实际应用造成严峻挑战。针对该问题,本文提出一种可同时对模型连接结构和权重位宽进行自适应联合优化的压缩方法。与已有组合式压缩不同,本文充分融合稀疏化和量化方法进行联合压缩训练,从而全面降低模型规模;采用层级自适应的稀疏度和数据表征位宽,缓解因固定压缩比导致的精度次优化问题。通过使用本文提出方法对VGG、ResNet和MobileNet在CIFAR-10数据集上的实验表明,精度损失分别为1.3%、2.4%和0.9%时,参数压缩率达到了143.0×、151.6×和19.7×;与12种典型压缩方法相比,模型存储资源的消耗降低了15.3×~148.5×。此外,在自建的遥感图像数据集上,该方法仍能在达到最高284.2×压缩率的同时保证精度损失不超过1.2%。Deep neural network compression methods with a single and fixed pattern are difficult to compress the model sufficiently due to the limitation of accuracy loss.As a result,the compressed model still needs to consume costly and limited storage resources when it is deployed,which is a significant barrier to its use in edge devices.To address this problem,this article proposes an adaptive joint compression method,which optimizes model structure and weight bit-width in parallel.Compared with the majority of existing combined compression methods,adequate fusion of sparsity and quantization methods is performed for joint compression training to reduce model parameter redundancy comprehensively.Meanwhile,the layer-wise adaptive sparse ratio and weight bit-width are designed to solve the sub-optimization problem of model accuracy and improve model accuracy loss due to the fixed compression ratio.Experimental results of VGG,ResNet,and MobileNet using the CIFAR-10 dataset show that the proposed method achieves 143.0×,151.6×,and 19.7×parameter compression ratios.The corresponding accuracy loss values are 1.3%,2.4%,and 0.9%,respectively.In addition,compared with 12 typical compression methods,the proposed method reduces the consumption of hardware memory resources by 15.3×~148.5×.In addition,the proposed method achieves maximum compression ratio of 284.2×whilemaintaining accuracy loss within limited range of 1.2%on the self-built remote sensing optical image dataset.

关 键 词：深度神经网络 模型压缩 联合优化 稀疏化 量化 

分 类 号：TP183[自动化与计算机技术—控制理论与控制工程] TH89[自动化与计算机技术—控制科学与工程]