基于训练数据动态分配的深度学习并行优化机制  

Deep learning parallel optimization mechanism based on dynamic distribution of training data

作　　者：颜子杰 陈孟强 吴维刚[1] YAN Zi-jie;CHEN Meng-qiang;WU Wei-gang(School of Data and Computer Science,Sun Yat-sen University,Guangzhou 510006,China)

机构地区：[1]中山大学数据科学与计算机学院,广东广州510006

出　　处：《计算机工程与科学》2018年第A01期141-144,共4页Computer Engineering & Science

基　　金：国家重点研发计划(2016YFB0200404);国家自然科学基金(U1711263)

摘　　要：基于MXNet框架,针对同步并行下参数同步耗时过长这一问题,提出了一种多机同步并行下的训练数据动态分配算法。基于计算节点的计算效率,每一次迭代后将动态调整节点需要处理的样本数据量。这样的机制使模型既能同步并行也降低了等待梯度更新的耗时。最后,利用天河二号超级计算机对此优化算法进行了对比实验,实验结果表明,所提出的优化机制达到了预期效果。To solve the time-consuming problem of collecting gradient updates under synchronous parallel training, we present a dynamic training data distribution algorithm under parallel synchronization of multiple machines. By calculating the computational efficiency of nodes, the amount of sample data that needs to be processed by nodes is dynamically assigned after each round of iteration. Such a mechanism allows the model to parallelize synchronously and reduce the waiting time it takes for gradient update. Finally, the mechanism is implemented via MXNet and evaluated at Tianhe-2 supercomputers. Experimental results show that the proposed optimization mechanism achieves expected results.

关 键 词：深度学习 数据分配 同步并行 并行训练 超级计算 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]