自然语言处理中的注意力机制研究综述  

Review of Attention Mechanism in Natural Language Processing

作　　者：石磊[1] 王毅[2] 成颖[2,3] 魏瑞斌 Shi Lei;Wang Yi;Cheng Ying;Wei Ruibin(School of Management Science and Engineering,Anhui University of Finance and Economics,Bengbu 233030,China;School of Information Management,Nanjing University,Nanjing 210023,China;School of Chinese Language and Literature,Shandong Normal University,Jinan 250014,China)

机构地区：[1](安徽财经大学管理科学与工程学院,蚌埠233030 [2](南京大学信息管理学院,南京210023 [3](山东师范大学文学院,济南250014

出　　处：《数据分析与知识发现》2020年第5期1-14,共14页Data Analysis and Knowledge Discovery

基　　金：国家社会科学基金重大项目“中国近现代文学期刊全文数据库建设与研究(1872-1949)”(项目编号:17ZDA276)的研究成果之一。

摘　　要：【目的】总结注意力机制在自然语言处理领域的衍化及应用规律。【文献范围】以"attention"和"注意力"为检索词,分别检索WoS、The ACM Digital Library、arXiv以及中国知网,时间跨度限定为2015年1月至2019年10月,制定标准人工筛选自然语言处理领域的文献,最终获得68篇相关文献。【方法】在深入分析文献的基础上,归纳注意力机制的通用形式,梳理其衍生类型,并基于数据对其在自然语言处理任务中的应用情况进行述评。【结果】注意力机制在自然语言处理中的应用集中于序列标注、文本分类、推理以及生成式任务,且任务和注意力机制的类型之间存在一定的适配规律。【局限】部分注意力机制和任务间的适配结论是通过模型整体表现数据间接得出的,不同注意力机制间的性能差异有待进一步研究。【结论】注意力机制的研究切实推进了自然语言处理的发展,但其作用机理尚未明了,提高其可解释性并使之更加接近人类的真实注意力是未来的研究方向。[Objective] This paper summarizes the evolution and application of attention mechanism in natural language processing. [Coverage] We searched"attention"with the title/topic fields of WoS, ACM Digital Library,arXiv and CNKI from January 2015 to October 2019. Then, we manually screened the topic literature in the field of natural language processing, and obtained 68 related papers. [Methods] We first summarized the general attention mechanism, and sorted out its derivations. Second, we thoroughly reviewed their applications in natural language processing tasks. [Results] The application of attention mechanism in natural language processing focused on sequence labeling, text classification, reasoning and generative tasks. There were adaptation rules between tasks and the various attention mechanisms. [Limitations] Some adaptations between the mechanisms and the tasks were obtained from the overall performance of the model. More research is needed to examine the performance of different mechanisms. [Conclusions] The study of attention mechanism has effectively promoted the development of natural language processing. However, the mechanism of action is not yet clear. Future research should focus on making attention mechanism closer to those of the human beings.

关 键 词：注意力机制 自注意力 机器翻译 机器阅读理解 情感分析 

分 类 号：TP391.1[自动化与计算机技术—计算机应用技术]