Low rank optimization for efficient deep learning:making a balance between compact architecture and fast training  

作　　者：OU Xinwei CHEN Zhangxin ZHU Ce LIU Yipeng 

机构地区：[1]School of Information and Communication Engineering,University of Electronic Science and Technology of China,Chengdu 611731,China

出　　处：《Journal of Systems Engineering and Electronics》2024年第3期509-531,F0002,共24页系统工程与电子技术（英文版）

基　　金：supported by the National Natural Science Foundation of China(62171088,U19A2052,62020106011);the Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China(ZYGX2021YGLH215,ZYGX2022YGRH005)。

摘　　要：Deep neural networks(DNNs)have achieved great success in many data processing applications.However,high computational complexity and storage cost make deep learning difficult to be used on resource-constrained devices,and it is not environmental-friendly with much power cost.In this paper,we focus on low-rank optimization for efficient deep learning techniques.In the space domain,DNNs are compressed by low rank approximation of the network parameters,which directly reduces the storage requirement with a smaller number of network parameters.In the time domain,the network parameters can be trained in a few subspaces,which enables efficient training for fast convergence.The model compression in the spatial domain is summarized into three categories as pre-train,pre-set,and compression-aware methods,respectively.With a series of integrable techniques discussed,such as sparse pruning,quantization,and entropy coding,we can ensemble them in an integration framework with lower computational complexity and storage.In addition to summary of recent technical advances,we have two findings for motivating future works.One is that the effective rank,derived from the Shannon entropy of the normalized singular values,outperforms other conventional sparse measures such as the?_1 norm for network compression.The other is a spatial and temporal balance for tensorized neural networks.For accelerating the training of tensorized neural networks,it is crucial to leverage redundancy for both model compression and subspace training.

关 键 词：model compression subspace training effective rank low rank tensor optimization efficient deep learning 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]