一种改进的分布式同步梯度下降法  

An Improved Distributed Synchronous Gradient Descent Method

作　　者：李梁[1] 王也 朱小飞 LI Liang;WANG Ye;ZHU Xiaofei(School of Computer Science and Engineering,Chongqing University of Technology,Chongqing 400054,China)

机构地区：[1]重庆理工大学计算机科学与工程学院,重庆400054

出　　处：《重庆理工大学学报（自然科学）》2021年第7期139-145,共7页Journal of Chongqing University of Technology：Natural Science

摘　　要：在深度学习领域中,模型的训练往往非常耗时,尤其是当模型较复杂时,分布式训练则是解决这个问题的一个主要方式。以往的案例中,用分布式训练神经网络能够得到非常好的加速效果,是因为采用了异步梯度下降法,但是这样会导致准确率下降。也有用同步梯度下降法来提升训练的准确率,但由于实际分布式的异构集群中各个计算节点的算力差距,会出现计算节点空闲等待的现象,使得模型的训练耗时十分不理想。采取改进同步梯度下降方法,通过设计新的样本分配机制,充分利用各个工作节点来加速模型训练。实验结果证明:所采取的方法在不降低准确率的情况下能够加快模型的训练速度。In the field of deep learning,the training of models is often time-consuming,especially when the model is more complex,distributed training is a major way to solve this problem.In previous cases,the distributed training of neural networks can achieve very good acceleration results because of the asynchronous gradient descent method,but this method will lead to the decline of accuracy due to the obsolete gradient parameters.It has also been proposed the synchronous gradient descent method to improve the accuracy of training,however,due to the gap between the computing nodes in the actual distributed heterogeneous cluster,the serious waiting mechanism makes the training of the model very time-consuming.In this paper,an improved synchronous gradient descent method is adopted.By designing a new sample allocation mechanism,each working node is fully utilized to accelerate the training of the model.The experimental results show that the method adopted in this paper can make the training of the model faster without reducing the accuracy.

关 键 词：深度学习 模型训练 分布式 梯度下降 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]