注意力机制综述  

Review of attention mechanism

作　　者：任欢 王旭光[1] REN Huan;WANG Xuguang(Department of Automation,North China Electric Power University,Baoding Hebei 071003,China)

机构地区：[1]华北电力大学自动化系,河北保定071003

出　　处：《计算机应用》2021年第S01期1-6,共6页journal of Computer Applications

摘　　要：现在注意力机制已广泛地应用在深度学习的诸多领域。基于注意力机制的结构模型不仅能够记录信息间的位置关系,还能依据信息的权重去度量不同信息特征的重要性。通过对信息特征进行相关与不相关的抉择建立动态权重参数,以加强关键信息弱化无用信息,从而提高深度学习算法效率同时也改进了传统深度学习的一些缺陷。从图像处理、自然语言处理、数据预测等不同应用方面介绍了一些与注意力机制结合的算法结构,并对近几年大火的基于注意力机制的transformer和reformer算法进行了综述。鉴于注意力机制的重要性,综述了注意力机制的研究发展,分析了注意力机制目前的发展现状并探讨了该机制未来可行的研究方向。Now the attention mechanism has been widely used in many fields of deep learning.The structural model based on the attention mechanism can not only record the positional relationship between information,but also measure the importance of different information features based on the weight of the information.Through the selection of relevant and irrelevant information features,dynamic weight parameters were established to strengthen key information and weaken useless information,thereby improving the efficiency of deep learning algorithms and improving some of the defects of traditional deep learning.Some algorithm structures combined with the attention mechanism were introduced from different application fields of image processing,natural language processing,and data prediction,and the attention mechanism-based transformer and reformer algorithms that have been popular in recent years were reviewed.In view of the importance of attention mechanism,the research development of attention mechanism was reviewed,the current development status of attention mechanism was analyzed,and the feasible research directions of this mechanism in the future were discussed.

关 键 词：注意力机制 深度学习 位置关系 信息特征 关键信息 TRANSFORMER REFORMER 

分 类 号：TP183[自动化与计算机技术—控制理论与控制工程]