主流大数据并行计算系统性能优化研究  

Research on performance optimization of mainstream big data parallel computing system

作　　者：吕亚荣 LV Yarong(Xi’an Mingde Institute of Technology,Xi’an 710000,China)

机构地区：[1]西安明德理工学院,西安710000

出　　处：《自动化与仪器仪表》2023年第8期100-104,共5页Automation & Instrumentation

基　　金：西安明德理工学院科研基金资助(2021XY01L07)。

摘　　要：为进一步提高大数据并行计算系统的性能,提出一种异步并行的计算部署方式。首先,改变以往的大数据节点主从结构,通过Ring-All Reduce对集群计算设备进行互联;然后提出该计算结构下的加速方式,并对异步并行中的梯度更新和异步通信进行设计。结果表明,在CPU的并行化技术和OpenBl AS库实验环境下,通过Cifar10数据集进行系统测试后,采用提出的方法很大程度上缩短了系统运行和训练的时间,也提高了并行计算的加速度和准确率。实际应用发现,将YOLOV4目标检测算法进行并行部署,其训练时间仅为6138 s,明显低于正常训练的26874 s,准确率和加速比也较正常训练高。由此说明,提出的主流大数据并行计算系统性能优化方法具备可行性。To further improve the performance of big data parallel computing system,an asynchronous parallel computing deployment method is proposed.First,change the previous master and slave structure of big data nodes,interconnect the cluster computing devices through Ring-All Reduce;then propose the acceleration mode under the computing structure,and design the gradient update and asynchronous communication in asynchronous parallel.The results show that under the parallelization technology of CPU and the OpenBl AS library experiment,the proposed method greatly reduces the time of system operation and training,and also improves the acceleration and accuracy of parallel computing.Practical application found that with the deployment of YOLOV4 target detection algorithm in parallel,the training time is only 6138s,which is significantly lower than the 26874s of normal training,and the accuracy and acceleration ratio are also higher than that of normal training.This shows that the proposed performance optimization method of the mainstream big data parallel computing system is feasible.

关 键 词：异步并行 梯度更新 YOLOV4目标检测 Ring-All Reduce 深度学习 

分 类 号：TP399[自动化与计算机技术—计算机应用技术]