Visual Topic Semantic Enhanced Machine Translation for Multi-Modal Data Efficiency  

作　　者：王超 蔡思佳 史北祥 崇志宏 Chao Wang;Si-Jia Cai;Bei-Xiang Shi;Zhi-Hong Chong(School of Computer Science and Engineering,Southeast University,Nanjing 210096,China;School of Architecture,Southeast University,Nanjing 210096,China)

机构地区：[1]School of Computer Science and Engineering,Southeast University,Nanjing 210096,China [2]School of Architecture,Southeast University,Nanjing 210096,China

出　　处：《Journal of Computer Science & Technology》2023年第6期1223-1236,共14页计算机科学技术学报（英文版）

基　　金：supported by the National Natural Science Foundation of China under Grant No.52178034.

摘　　要：The scarcity of bilingual parallel corpus imposes limitations on exploiting the state-of-the-art supervised translation technology.One of the research directions is employing relations among multi-modal data to enhance perfor-mance.However,the reliance on manually annotated multi-modal datasets results in a high cost of data labeling.In this paper,the topic semantics of images is proposed to alleviate the above problem.First,topic-related images can be auto-matically collected from the Internet by search engines.Second,topic semantics is sufficient to encode the relations be-tween multi-modal data such as texts and images.Specifically,we propose a visual topic semantic enhanced translation(VTSE)model that utilizes topic-related images to construct a cross-lingual and cross-modal semantic space,allowing the VTSE model to simultaneously integrate the syntactic structure and semantic features.In the above process,topic similar texts and images are wrapped into groups so that the model can extract more robust topic semantics from a set of similar images and then further optimize the feature integration.The results show that our model outperforms competitive base-lines by a large margin on the Multi30k and the Ambiguous COCO datasets.Our model can use external images to bring gains to translation,improving data efficiency.

关 键 词：multi-modal machine translation visual topic semantics data efficiency 

分 类 号：TP391.2[自动化与计算机技术—计算机应用技术]