Continuous Sign Language Recognition Based on Spatial-Temporal Graph Attention Network  

作　　者：Qi Guo Shujun Zhang Hui Li 

机构地区：[1]College of Information Science and Technology,Qingdao University of Science and Technology,Qingdao,266061,China

出　　处：《Computer Modeling in Engineering & Sciences》2023年第3期1653-1670,共18页工程与科学中的计算机建模（英文）

基　　金：supported by the Key Research&Development Plan Project of Shandong Province,China(No.2017GGX10127).

摘　　要：Continuous sign language recognition(CSLR)is challenging due to the complexity of video background,hand gesture variability,and temporal modeling difficulties.This work proposes a CSLR method based on a spatialtemporal graph attention network to focus on essential features of video series.The method considers local details of sign language movements by taking the information on joints and bones as inputs and constructing a spatialtemporal graph to reflect inter-frame relevance and physical connections between nodes.The graph-based multihead attention mechanism is utilized with adjacent matrix calculation for better local-feature exploration,and short-term motion correlation modeling is completed via a temporal convolutional network.We adopted BLSTM to learn the long-termdependence and connectionist temporal classification to align the word-level sequences.The proposed method achieves competitive results regarding word error rates(1.59%)on the Chinese Sign Language dataset and the mean Jaccard Index(65.78%)on the ChaLearn LAP Continuous Gesture Dataset.

关 键 词：Continuous sign language recognition graph attention network bidirectional long short-term memory connectionist temporal classification 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]