基于混合自注意力机制的神经机器翻译  

Hybrid Self-Attention Network for Neural Machine Translation

作　　者：宋恺涛 陆建峰[1] SONG Kaitao;LU Jianfeng(School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing,Jiangsu 210094,China)

机构地区：[1]南京理工大学计算机科学与工程学院,江苏南京210094

出　　处：《中文信息学报》2023年第9期38-45,共8页Journal of Chinese Information Processing

摘　　要：编码器-解码器结构是神经机器翻译最常用的一种框架,许多新型结构都基于此框架进行设计以改善翻译性能。其中,深度自注意力网络是非常出色的一种网络结构,其利用了自注意力机制来捕获全局的语义信息。然而,这种网络却不能有效地区分每个单词的相对位置,例如,依赖单词究竟位于目标单词的左边还是右边,也不能够捕获当前单词的局部语义。为了缓解这类问题,该文提出了一种新型的注意力机制,叫做混合注意力机制。该机制包含了对自注意力网络设计的多种不同的特定掩码来获取不同的语义信息,例如,全局和局部信息,以及左向或者右向信息。最后,该文提出了一个压缩门来融合不同类型的自注意力网络。在三个机器翻译数据集上的实验结果表明,该文方法能够取得比深度自注意力网络更好的结果。For Neural Machine Translation(NMT),Transformer is one of the most promising structures,which can leverage the self-attention mechanism to capture the semantic dependency from global view.However,it cannot distinguish the relative position of different tokens very well,such as the tokens located at the left or right of the current token,and cannot focus on the local information around the current token either.To alleviate these problems,we propose a novel attention mechanism named Hybrid Self-Attention Network(HySAN)with specific-designed masks for self-attention network to extract various semantic,such as the global/local information,the left/right part context.Finally,a squeeze gate is introduced to combine different kinds of SANs for fusion.Experimental results on three machine translation tasks show that the proposed appraoch outperforms the Transformer baseline significantly.

关 键 词：自注意力 神经机器翻译 深度神经网络 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]