深度学习中的预条件动量梯度算法  

Preconditioned momentum gradient algorithm in deep learning

作　　者：张亮亮[1] 喻高航 ZHANG Liangliang;YU Gaohang(School of Sciences,Hangzhou Dianzi University,Hangzhou Zhejiang 310018,China)

机构地区：[1]杭州电子科技大学理学院,浙江杭州310018

出　　处：《杭州电子科技大学学报（自然科学版）》2021年第5期81-87,共7页Journal of Hangzhou Dianzi University：Natural Sciences

基　　金：国家自然科学基金资助项目(11661007);浙江省自然科学基金资助项目(LD19A010002);杭州电子科技大学研究生科研创新基金资助项目(CXJJ2019032)。

摘　　要：结合“比例积分微分”优化控制器(Proportional-Integral-Derivative Controller,PID)与拟牛顿方法,提出一种加速神经网络训练的一类预条件动量梯度算法。首先,采用拟牛顿条件产生预条件因子,然后,在迭代过程中与动量方法相结合,提高算法效率的同时,克服原始动量梯度算法的超调问题,并证明了算法在目标函数强凸时的全局收敛性。最后,通过数值实验验证了算法的有效性。Deep learning usually requires solving large-scale optimization problem.Based on the traditional Proportional-Integral-Derivative Controller(PID),combined with the Quasi-Newton Method,this paper proposes a kind of preconditioned momentum algorithm to accelerate neural network training.It can effectively overcome the overshoot phenomenon of the original momentum method.Moreover,it can also improve greatly the efficiency of the algorithm due to the usage of the preconditioning factors.When the objective function is strongly convex,the global convergence of the algorithm can be guaranteed.Finally,numerical experiments verify the effectiveness of the proposed algorithm.

关 键 词：动量梯度算法 PID算法 拟牛顿方程 超调问题 

分 类 号：O224[理学—运筹学与控制论]