基于改进MobileNet网络的AR辅助手语字母识别方法  

AR-assisted Sign Language Letter Recognition Method Based on Improved MobileNet Network

作　　者：刘春宏 王松[1] 王赋攀[1] 唐文生 裴云强 田东生 吴亚东 Liu Chunhong;Wang Song;Wang Fupan;Tang Wensheng;Pei Yunqiang;Tian Dongsheng;Wu Yadong(School of Computer Science&Technology,Southwest University of Science and Technology,Mianyang 621010,China;School of Computer Science&Engineering,Sichuan University of Science and Engineering,Zigong 643002,China)

机构地区：[1]西南科技大学计算机科学与技术学院,四川绵阳621010 [2]四川轻化工大学计算机科学与工程学院,四川自贡643002

出　　处：《系统仿真学报》2023年第6期1308-1321,共14页Journal of System Simulation

基　　金：国家自然科学基金面上项目(61872304,61802320);四川省自然科学基金(2022NSFSC0961);西南科技大学博士基金(19zx7144);西南科技大学素质类教改(青年发展研究)专项(20szjg17)。

摘　　要：针对手语手势姿态待规范、识别率低的问题,提出一种AR辅助手语字母识别算法MSMobileNet。设计多尺度卷积模块提取底层特征,增强网络的特征提取能力;利用ELU激活函数来保留更全面的负值特征信息;结合适用于Web的轻量级MobileNet模型,提高面向移动AR应用的手语字母识别准确率和实时性。实验结果表明:MS-MobileNet在数据集ASL-M、NUS-II和Creative Senz3D上识别准确率较原模型分别提高了2.58%、5.32%和3.04%。基于MS-MobileNet网络设计一套WebAR辅助的手语字母协同交互系统,经评估测试,用户平均参与度达到8.2分,单次识别耗时低于0.115 s,能较好地满足用户沉浸式的实时手语字母交互需求。An AR-assisted sign language letter recognition algorithm MS-MobileNet is proposed for the problems of sign language gestures needing to be standardized and low recognition rate.A multi-scale convolution module is designed to extract the low-level features and enhance the feature extraction ability.ELU activation function is used to retain the negative feature information,which combined with a lightweight MobileNet model for the web to improve the recognition accuracy and real-time performance for mobile AR applications.Test results show that compared with the original model,the recognition accuracy of MS-MobileNet on the datasets ASL-M,NUS-II and Creative Senz3D is improved by 2.58%,5.32% and 3.04%,respectively.Based on improved network,a WebAR-assisted sign language collaborative interaction system is designed.After the evaluation test,the average user participation rate reached 8.2 points,and the single recognition time is less than 0.115 s.User's needs for immersive realtime sign language letter interaction is better met.

关 键 词：手语字母识别 MobileNet 多尺度卷积 WebAR 协同交互 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]