A revisit to MacKay algorithm and its application to deep network compression  

作　　者：Chune LI Yongyi MAO Richong ZHANG Jinpeng HUAI 

机构地区：[1]School of Computer Science and Engineering,Beihang University,Beijing,100191,China [2]School of Electrical Engineering and Computer Science,University of Ottawa,Ottawa,K1N6N5,Canada

出　　处：《Frontiers of Computer Science》2020年第4期39-54,共16页中国计算机科学前沿（英文版）

基　　金：This work was supported partly by China Scholarship Council(201706020062);by China 973 program(2015CB358700);by the National Natural Science Foundation of China(Grant Nos.61772059,61421003);by the Beijing Advanced Innovation Center for Big Data and Brain Computing(BDBC);State Key Laboratory of Software Development Environment(SKLSDE-2018ZX-17).

摘　　要：An iterative procedure introduced in MacKay’s evidence framework is often used for estimating the hyperparameter in empirical Bayes.Together with the use of a particular form of prior,the estimation of the hyperparameter reduces to an automatic relevance determination model,which provides a soft way of pruning model parameters.Despite the effectiveness of this estimation procedure,it has stayed primarily as a heuristic to date and its application to deep neural network has not yet been explored.This paper formally investigates the mathematical nature of this procedure and justifies it as a well-principled algorithm framework,which we call the MacKay algorithm.As an application,we demonstrate its use in deep neural networks,which have typically complicated structure with millions of parameters and can be pruned to reduce the memory requirement and boost computational efficiency.In experiments,we adopt MacKay algorithm to prune the parameters of both simple networks such as LeNet,deep convolution VGG-like networks,and residual netowrks for large image classification task.Experimental results show that the algorithm can compress neural networks to a high level of sparsity with little loss of prediction accuracy,which is comparable with the state-of-the-art.

关 键 词：deep learning MacKay algorithm model compression neural network 

分 类 号：TP393[自动化与计算机技术—计算机应用技术]