基于Transformer和多模态对齐的非自回归手语翻译技术研究  

Non-Autoregressive Sign Language Translation Technology Based on Transformer and Multimodal Alignment

作　　者：邵舒羽 杜垚 范晓丽 SHAO Shuyu;DU Yao;FAN Xiaoli(School of Logistics,Beijing Wuzi University,Beijing 101149,China;School of Automation Science and Electrical Engineering,Beihang University,Beijing 100191,China;Air force medical center,PLA,Beijing 101142,China)

机构地区：[1]北京物资学院物流学院,北京101149 [2]北京航空航天大学自动化科学与电气工程学院,北京100191 [3]空军特色医学中心,北京100142

出　　处：《电子与信息学报》2024年第7期2932-2941,共10页Journal of Electronics & Information Technology

基　　金：国家自然科学基金(8210072143);北京市教委科技计划青年项目(KM202210037001)。

摘　　要：为了解决多模态数据的对齐及手语翻译速度较慢的问题,该文提出一个基于自注意力机制模型Transformer的非自回归手语翻译模型(Trans-SLT-NA),同时引入了对比学习损失函数进行多模态数据的对齐,通过学习输入序列(手语视频)和目标序列(文本)的上下文信息和交互信息,实现一次性地将手语翻译为自然语言。该文所提模型在公开数据集PHOENIX-2014T(德语)、CSL(中文)和How2Sign(英文)上进行实验评估,结果表明该文方法相比于自回归模型翻译速度提升11.6~17.6倍,同时在双语评估辅助指标(BLEU-4)、自动摘要评估指标(ROUGE)指标上也接近自回归模型。To address the challenge of aligning multimodal data and improving the slow translation speed in sign language translation,a Transformer Sign Language Translation Non-Autoregression(Trans-SLT-NA)is proposed in this paper,which utilizes a self-attention mechanism.Additionally,it incorporates a contrastive learning loss function to align the multimodal data.By capturing the contextual and interaction information between the input sequence(sign language videos)and the target sequence(text),the proposed model is able to perform sign language translation to natural language in s single step.The effectiveness of the proposed model is evaluated on publicly available datasets,including PHOENIX-2014-T(German),CSL(Chinese)and How2Sign(English).Results demonstrate that the proposed method achieves a significant improvement in translation speed,with a speed boost ranging from 11.6 to 17.6 times compared to autoregressive models,while maintaining comparable performance in terms of BiLingual Evaluation Understudy(BLEU-4)and Recall-Oriented Understudy for Gisting Evaluation(ROUGE)metrics.

关 键 词：手语翻译 自注意力机制 非自回归翻译 深度学习 多模态数据对齐 

分 类 号：TN108.4[电子电信—物理电子学] TP391[自动化与计算机技术—计算机应用技术]