基于注意力机制的复杂背景连续手语识别  

Continuous Sign Language Recognition in Complex Background Based on Attention Mechanism

作　　者：杨光义[1] 丁星宇 高毅 胡晶欣 张洪艳[2] YANG Guangyi;DING Xingyu;GAO Yi;HU Jingxin;ZHANG Hongyan(Electronic Information School,Wuhan University,Wuhan 430072,Hubei,China;State Key Laboratory of Information Engineering in Surveying,Mapping and Remote Sensing,Wuhan University,Wuhan 430079,Hubei,China)

机构地区：[1]武汉大学电子信息学院,湖北武汉430072 [2]武汉大学测绘遥感信息工程国家重点实验室,湖北武汉430079

出　　处：《武汉大学学报（理学版）》2023年第1期97-105,共9页Journal of Wuhan University:Natural Science Edition

基　　金：国家自然科学基金面上项目(42071322);湖北省杰出青年基金(2020CFA053);武汉市应用基础前沿项目(2020010601012184)。

摘　　要：提出一种基于注意力机制的连续手语识别算法ACN(attention-based 3D convolutional neural network),能够识别复杂背景下的连续手语。该算法首先利用背景去除模块,对包含复杂背景的手语视频进行预处理;然后,通过基于空间注意力机制的3D-ResNet(3D residual convolutional neural network)提取时空融合信息;最后,采用结合时间注意力机制的长短期记忆(long short-term memory,LSTM)网络进行序列学习,得到最终的识别结果。算法在大规模中国连续手语数据集CSL100上表现优异;在面向不同复杂背景的情况下,算法表现出良好的泛化性能,模型引入的时空注意力机制是切实有效的。In this work, an attention-based 3D convolutional neural network(ACN) is proposed for continuous sign language recognition in complex background. Firstly, the sign language video containing complex background is preprocessed with the background removal module. Then, the spatio-temporal fusion information is extracted by 3D-ResNet(3D residual convolutional neural network) based on spatial attention mechanism. Finally, the long short-term memory(LSTM) network combined with the time attention mechanism is used for sequence learning to obtain the final recognition result. Extensive experiments show that the algorithm performs well on the large-scale Chinese continuous sign language dataset CSL100. The algorithm shows good generalization performance facing different complex backgrounds, and the spatio-temporal attention mechanism introduced by the model is effective.

关 键 词：连续手语识别 复杂背景 注意力机制 长短期记忆 

分 类 号：TP391.2[自动化与计算机技术—计算机应用技术]