深度神经网络在线训练硬件加速器的数据量化综述  

A review of data quantization of deep neural network online training hardware accelerator

作　　者：汪航[1] 李宝婷 张旭翀[2] 李红光[3] 杜明超 孙宏滨[2] WANG Hang;LI Baoting;ZHANG Xuchong;LI Hongguang;DU Mingchao;SUN Hongbin(School of Microelectronics,Xi'an Jiaotong University,Xi'an 710049,China;College of Artificial Intelligence,Xi'an Jiaotong University,Xi'an 710049,China;Xi'an Institute of Applied Optics,Xi'an 710065,China)

机构地区：[1]西安交通大学微电子学院,陕西西安710049 [2]西安交通大学人工智能学院,陕西西安710049 [3]西安应用光学研究所,陕西西安710065

出　　处：《微电子学与计算机》2024年第3期1-11,共11页Microelectronics & Computer

基　　金：国家自然科学基金(62204200,62004157);中央高校基本科研业务费专项(xhj032021005-01)。

摘　　要：随着算法和数据的爆炸式增长,深度神经网络(Deep Neural Network, DNN)逐渐在实际应用中扮演愈发重要的角色。然而,真实场景中的数据与线下训练数据之间往往并不满足独立同分布假设,导致预训练DNN模型在实际应用中性能严重下降。所以,在资源供给相对有限的平台上进行DNN模型在线训练成为其有效应用的保证。为了满足真实场景对DNN模型质量与速度的多维度性能要求,如何在保证算法精度的同时显著降低计算复杂度是在此类应用中部署DNN的关键。数据量化是降低计算复杂度的主流优化技术之一,能够通过降低模型参数、中间值等数据的位宽来减少硬件加速器的资源耗费。因此,从软件和硬件两个方面对深度神经网络训练加速器中关于数据量化的研究进行总结。对国内外最新发表的相关文献进行归纳总结。首先,从软件的角度总结了不同的量化方法,包括简单映射数据量化和复杂映射数据量化;其次,从硬件的角度总结了DNN加速器对网络在线训练各计算步骤的量化支持;再次,阐述了数据量化对加速器设计的影响,包括存储单元和计算单元;最后,对本领域的研究进行总结,并展望了未来本领域的发展方向。文章提出的分类方法有助于对之前的DNN加速器在数据量化方面的工作进行分类。With the explosive growth of algorithms and data,the Deep Neural Network(DNN)gradually plays an increasingly important role in practical applications.However,it is difficult for the real scene and offline training data to meet the assumption of independent and identical distribution,resulting in a serious decline in the performance of the pretraining DNN model in practical applications.Therefore,online training of DNN model on the platform with relatively limited resources becomes the guarantee of its effective application.Hence,it is necessary to significantly reduce the computational complexity while ensuring the accuracy. Data quantization is one of the mainstream optimization technologies to reduce computational complexity. Accordingly, we summarize the researches on data quantization of DNN online training accelerator. Firstly, data quantization based on direct fixed-point representation and complex mapping are summarized from the perspective of software. Secondly, the quantization of DNN accelerator for each training step is summarized from the perspective of hardware. Then, the influence of data quantization on accelerator design is described, including memory unit and processing unit. Finally, the researches in this field are summarized and the future development directions of this field are prospected. The classification method proposed in this paper is helpful to classify the previous work of DNN accelerator in data quantization.

关 键 词：深度神经网络 在线训练 加速器 数据量化 

分 类 号：TP402[自动化与计算机技术]