融合图文预训练的汉越多模态神经机器翻译  

Chinese-Vietnamese multimodal neural machine translation with integrated image-text pre-training

作　　者：韦浩翔 高盛祥[1,2] 余正涛 王晓聪[1,2] Wei Haoxiang;Gao Shengxiang;Yu Zhengtao;Wang Xiaocong(Faculty of Information Engineering and Automation,Kunming University of Science and Technology,Kunming 650500,China;Yunnan Key Laboratory of Artificial Intelligence,Kunming 650500,China)

机构地区：[1]昆明理工大学信息工程与自动化学院,云南昆明650500 [2]云南省人工智能重点实验室,云南昆明650500

出　　处：《电子技术应用》2024年第12期48-54,共7页Application of Electronic Technique

基　　金：国家自然科学基金(U21B2027);云南省重点研发计划(202401BC070021);云南省科技人才与平台计划(202105AC160018)。

摘　　要：由于汉语和越南语之间存在显著的语法差异及语料稀缺,汉越神经机器翻译任务面临名词翻译不准确的挑战。提出了一种新颖的多模态神经机器翻译方法,该方法融合了文本预训练模型和视觉语言联合预训练模型。通过文本预训练模型,能够捕获深层的语言结构和语义;而视觉语言联合训练模型则提供了与文本相关联的视觉上下文,这有助于模型更准确地理解和翻译名词。两种模型通过一个简洁高效的映射网络结合,并通过Gumbel门控模块动态地整合多模态信息,以优化翻译输出。在汉越及越汉翻译任务中,该方法相比传统Transformer模型分别提升了7.13和4.27的BLEU值。Due to significant grammatical differences and a scarcity of linguistic resources between Chinese and Vietnamese,the task of Chinese-Vietnamese neural machine translation faces challenges in the accurate translation of nouns.This paper proposes a novel multimodal neural machine translation method that integrates a text-based pre-trained model with a visual-linguistic joint pre-training model.The text-based model captures deep linguistic structures and semantics,while the visual-linguistic joint train-ing model provides visual context related to the text,which helps the model understand and translate nouns more accurately.The two models are combined through a streamlined and efficient mapping network and dynamically integrate multimodal information via a Gumbel gating module to optimize translation outputs.In both Chinese-Vietnamese and Vietnamese-Chinese translation tasks,this method has achieved improvements of 7.13 and 4.27 BLEU points,respectively,compared to the traditional Trans-former model.

关 键 词：汉越神经机器翻译 视觉-语言联合预训练 多模态 注意力 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]