卷积神经网络并行训练的优化研究  

Research on Optimization of Parallel Training for Convolution Neural Network

作　　者：李相桥 李晨[1] 田丽华[1] 张玉龙[1] LI Xiang-qiao;LIChen;TIAN Li-hua;ZHANG Yu-long(School of Software Engineering,Xi＇ an Jiaotong University,Xi＇ an 710049,China;AVIC Xi＇an Flight Automatic Control Research Institute,Xi＇ an 710065,China)

机构地区：[1]西安交通大学软件学院,陕西西安710049 [2]中航工业西安飞行自动控制研究所,陕西西安710065

出　　处：《计算机技术与发展》2018年第8期12-16,共5页Computer Technology and Development

基　　金：国家自然科学基金(61403302);中央高校基本科研业务费专项资金(XJJ2016029)

摘　　要：由于卷积神经网络模型结构复杂且计算量大,实际应用中一般采用多GPU的方式对其进行并行训练,快速地完成卷积神经网络的快速训练。为了提高卷积神经网络的并行训练效率,同时解决在并行训练时通讯缓慢以及数据加载时等待的问题,提出参数通讯以及数据加载两个方面的优化策略。在参数通讯优化方面,将梯度计算和参数通讯同时执行,利用计算时间来覆盖通讯时间。通过改变通讯方式,利用归约和冗余通讯方式减少参数通讯时的同步等待时间。利用预加载和异步拷贝的方式将数据提前加载并拷贝到GPU显存空间,减少数据加载带来的时间消耗。实验结果表明,优化后的方法能够有效地提高卷积神经网络的并行训练效率。As the model of convolution neural network is complex in structure and large in computation,the parallel training is generally carried out in the way of multi-GPU in practical application,and the rapid training of convolution neural network is completed quickly. In order to improve the parallel training efficiency of convolution neural network and solve the problem of slow communication and the waiting caused by data loading in parallel training,we propose two aspects of optimization strategy in parameter communication and data loading. In the aspect of the parameter communication optimization,the gradient calculation and the parameter communication are executed simultaneously,and the computation time is utilized to cover the communication time. By changing the communication mode,the synchronous waiting time of communication is reduced by means of reduction and redundancy communication. Using preload and asynchronous copying,the data is loaded and copied to the GPU memory space in advance to reduce the time consumption of data loading. The experiment shows that the optimized method can effectively improve the parallel training efficiency of convolution neural network.

关 键 词：卷积神经网络 数据并行 通讯优化 数据加载优化 

分 类 号：TP183[自动化与计算机技术—控制理论与控制工程]