结合注意力机制的深度神经网络综述  

A Review of Deep Neural Networks Combined with Attention Mechanism

作　　者：皇甫晓瑛 钱惠敏[1] 黄敏[1] HUANGFU Xiao-ying;QIAN Hui-min;HUANG Min(College of Energy and Electric Engineering,Hohai University,Nanjing 211100,China)

机构地区：[1]河海大学能源与电气学院,江苏南京211100

出　　处：《计算机与现代化》2023年第2期40-49,57,共11页Computer and Modernization

基　　金：国家自然科学基金资助项目(61573001)。

摘　　要：注意力机制已成为改进神经网络学习能力的研究热点之一。鉴于注意力机制受到的广泛关注,本文旨在从注意力机制的分类、与深度神经网络的结合方式,以及在自然语言处理和计算机视觉领域的具体应用3个方面对深度神经网络中的注意力机制给出较全面的分析和阐述。具体地,分析比较了软注意力、硬注意力和自注意力这3种机制的优缺点;并分别讨论了递归神经网络和卷积神经网络中结合注意力机制的常用方式及其代表性模型结构;然后,以自然语言处理、计算机视觉领域为例,说明了其应用情况;最后,分析了注意力机制的发展趋势,期望为后续研究提供线索和方向。Attention mechanism has become one of the research hotspots in improving the learning ability of deep neural network.In view of the wide attention paid to the attention mechanism,this paper aims to give a comprehensive analysis and elaboration of attention mechanism in deep neural network from three aspects:the classification of attention mechanism,the way of combining with deep neural network,and the specific applications in natural language processing and computer vision.Specifically,atten⁃tion mechanism has been divided into soft attention mechanism,hard attention mechanism and self-attention mechanism,and their advantages and disadvantages are compared.Then,the common ways of combining attention mechanism in recursive neural network and convolutional neural network are discussed respectively,and the representative model structures of each way are given.After that,the applications of attention mechanism in natural language processing and computer vision are illustrated.Fi⁃nally,several future developments of attention mechanism are illustrated expecting to provide clues and directions for subsequent researches.

关 键 词：注意力机制 深度学习 神经网络 注意力模型 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]