融合自适应图卷积与Transformer序列模型的中文手语翻译方法  

Chinese sign language translation method fusing adaptive graph convolution and Transformer sequence model

作　　者：应捷[1] 徐文成 杨海马[1] 刘瑾[2] 郑乐芊 Ying Jie;Xu Wencheng;Yang Haima;Liu Jin;Zheng Leqian(School of Optoelectronic Information&Computer Engineering,University of Shanghai for Science&Technology,Shanghai 200093,China;School of Electronic&Electrical Engineering,Shanghai University of Engineering Science,Shanghai 201620,China)

机构地区：[1]上海理工大学光电信息与计算机工程学院,上海200093 [2]上海工程技术大学电子电气工程学院,上海201620

出　　处：《计算机应用研究》2023年第5期1589-1594,1600,共7页Application Research of Computers

基　　金：国家自然科学基金资助项目(62172280)。

摘　　要：针对手语翻译方法所存在的动作特征提取以及时序翻译方面存在的问题,提出一种融合自适应图卷积AGCN与Transformer时序模型的AGCN-T手语翻译网络。自适应图卷积网络用于学习手语动作中骨骼节点的交互空间依赖信息;Transformer时序模块捕捉手语动作序列的时间关系特征信息并将其翻译成可理解的手语内容。此外,在预处理部分,提出了一种移动窗口的关键帧提取算法,并用MediaPipe姿态估计算法对关键帧图像序列进行骨架提取。实验表明,该方法在大型中文连续手语数据集CCSL的词错率达到了3.75%,精度为97.87%,优于其他先进的手语翻译方法。In order to solve the problems of feature extraction and temporal translation in sign language translation methods,this paper proposed an AGCN-T sign language translation network that combined AGCN and Transformer temporal model.The AGCN module learnt the interactive spatial dependency information of skeleton nodes in sign language actions.The Transformer temporal module captured the time relationship feature information of the sign language action sequence and translated it into understandable sign language semantic information.In addition,in the preprocessing part,this paper proposed a key frame extraction algorithm for moving windows,and used MediaPipe pose estimation algorithm to extract the skeleton of key frame image sequences.The experimental results show that the proposed method achieves a word error rate of 3.75%and an accuracy of 97.87%in the large-scale Chinese continuous sign language dataset CCSL,which is superior to other advanced sign language translation methods.

关 键 词：手语翻译 自适应图卷积 Transformer时序模型 关键帧提取 姿态估计 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]