Hand-aware graph convolution network for skeleton-based sign language recognition  

作　　者：Juan Song Huixuechun Wang Jianan Li Jian Zheng Zhifu Zhao Qingshan Li 

机构地区：[1]School of Computer Science and Technology,Xidian University,Xi'an 710126,China [2]School of Artificial Intelligence Engineering,Xidian University,Xi'an 710126,China

出　　处：《Journal of Information and Intelligence》2025年第1期36-50,共15页信息与智能学报(英文)

基　　金：supported by Young Scientists Fund of the National Natural Science Foundation of China(62202356,62302373);Fundamental Research Funds for the Central Universities(ZYTS24092,QTZX24085).

摘　　要：Skeleton-based sign language recognition(SLR)is a challenging research area mainly due to the fast and complex hand movement.Currently,graph convolution networks(GCNs)have been employed in skeleton-based SLR and achieved remarkable performance.However,existing GCN-based SLR methods suffer from a lack of explicit attention to hand topology which plays an important role in the sign language representation.To address this issue,we propose a novel hand-aware graph convolution network(HA-GCN)to focus on hand topological relationships of skeleton graph.Specifically,a hand-aware graph convolution layer is designed to capture both global body and local hand information,in which two sub-graphs are defined and incorporated to represent hand topology information.In addition,in order to eliminate the over-fitting problem,an adaptive DropGraph is designed in construction of hand-aware graph convolution block to remove the spatial and temporal redundancy in the sign language representation.With the aim to further improve the performance,the joints information,bones,together with their motion information are simultaneously modeled in a multi-stream framework.Extensive experiments on the two open-source datasets,AUTSL and INCLUDE,demonstrate that our proposed algorithm outperforms the state-of-the-art with a significant margin.Our code is available at https://github.com/snorlaxse/HA-SLR-GCN.

关 键 词：Sign language recognition Graph convolutional network Hand-aware graphs Skeleton data Multi-stream fusion 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]