Multi-scale context-aware network for continuous sign language recognition  

作　　者：Senhua XUE Liqing GAO Liang WAN Wei FENG 

机构地区：[1]College of Intelligence and Computing,Tianjin University,Tianjin 300354,China

出　　处：《虚拟现实与智能硬件（中英文）》2024年第4期323-337,共15页Virtual Reality & Intelligent Hardware

基　　金：Supported by the National Natural Science Foundation of China(62072334).

摘　　要：The hands and face are the most important parts for expressing sign language morphemes in sign language videos.However,we find that existing Continuous Sign Language Recognition(CSLR)methods lack the mining of hand and face information in visual backbones or use expensive and time-consuming external extractors to explore this information.In addition,the signs have different lengths,whereas previous CSLR methods typically use a fixed-length window to segment the video to capture sequential features and then perform global temporal modeling,which disturbs the perception of complete signs.In this study,we propose a Multi-Scale Context-Aware network(MSCA-Net)to solve the aforementioned problems.Our MSCA-Net contains two main modules:(1)Multi-Scale Motion Attention(MSMA),which uses the differences among frames to perceive information of the hands and face in multiple spatial scales,replacing the heavy feature extractors;and(2)Multi-Scale Temporal Modeling(MSTM),which explores crucial temporal information in the sign language video from different temporal scales.We conduct extensive experiments using three widely used sign language datasets,i.e.,RWTH-PHOENIX-Weather-2014,RWTH-PHOENIX-Weather-2014T,and CSL-Daily.The proposed MSCA-Net achieve state-of-the-art performance,demonstrating the effectiveness of our approach.

关 键 词：Continuous sign language recognition Multi-scale motion attention Multi-scale temporal modeling 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]