基于SVD的DNN裁剪方法和重训练  

SVD-based DNN pruning and retraining

作　　者：邢安昊 张鹏远[1] 潘接林[1] 颜永红[1] 

机构地区：[1]中国科学院声学研究所,语言声学与内容理解重点实验室,北京100190

出　　处：《清华大学学报（自然科学版）》2016年第7期772-776,共5页Journal of Tsinghua University(Science and Technology)

基　　金：国家自然科学基金资助项目(11461141004,91120001,61271426);国家“八六三”高技术项目(2012AA012503);中国科学院战略性先导科技专项(XDA06030100,XDA06030500);中国科学院重点部署项目(KGZD-EW-103-2)

摘　　要：深层神经网络(DNN)的参数量巨大,限制了其在一些计算资源受限或是注重速度的应用场景中的应用。为了降低DNN参数量,有学者提出利用奇异值分解(SVD)对DNN进行裁剪,然而其方法缺乏自适应性,因为它会从所有隐层裁减掉同样数量的奇异值。该文提出了一种基于奇异值比率裁剪因子(singular rate pruning factor,SRPF)的DNN裁剪方法。该方法以数据驱动的方式分别为DNN的各个隐层计算出SRPF,然后以不同的裁剪因子对各隐层进行裁剪,这充分利用了各隐层权值矩阵的奇异值分布特性。与固定数量裁剪法相比,该方法具有自适应性。实验表明:在同样裁剪力度下,该方法给DNN造成的性能损失更小。另外,该文还提出了一种适合裁剪后的DNN的重训练方法。Deep neural networks （DNN） have many parameters, which restricts the use of DNN in scenarios with limited computing resources or when speed is a priority. Some researchers have proposed to prune the DNN using singular value decomposition （SVD）. However, this method lacks adaptivity as it prunes the same number of singular values in all the hidden DNN layers. A singular rate pruning factor （SRPF） based DNN pruning method is given here. This method first separately calculates the SRPFs for each hidden layer based on the data with every layer then pruned using different pruning factors. This method makes full use of the distribution traits of the singular values in each hidden layer. This method is more adaptive than pruning a fixed portion of singular values with experiments showing that a DNN pruned with this method performs better. A retraining method is also given which adapts to the pruned DNN.

关 键 词：语音识别 深层神经网络(DNN) 奇异值分解(SVD) 

分 类 号：TN912.34[电子电信—通信与信息系统]