基于YOLO V5和Transformer的实时手语识别和翻译方法研究  

Research on Real time Sign Language Recognition and Translation Method Based on YOLO V5 and Transformer

作　　者：冯文静 王岩 张天宇 孙嘉玥 邓增辉 FENG Wenjing;WANG Yan;ZHANG Tianyu;SUN Jiayue;DENG Zenghui(Zhengzhou University of Science and Technology,Zhengzhou 450064,China)

机构地区：[1]郑州科技学院,郑州450064

出　　处：《移动信息》2024年第10期316-319,共4页Mobile Information

基　　金：河南省教育厅2023年大学生创新创业训练计划项目:基于ST-GNN和Transformer的手语翻译系统的设计与实现(202312746002);郑州科技学院2023年大学生创新创业训练计划项目:基于AI的手语识别以及翻译的研究与实现(DC202302)。

摘　　要：针对手语识别和翻译的需求,文中首先结合OpenCV和Mediapipe hands模型,制作了一个包含多种手势、不同角度、明暗区域、简单和复杂背景等特定环境下的手语数据集,该数据集共计3000张图像。其次,为了实现更准确和实时的手语识别和翻译,选用YOLO V5模型进行了手语识别检测。在测试集上,平均准确率mAP(0.5)达99.1%。然后,结合Transformer模型进行手语孤立词连接,实现手语句子的生成。这些技术的应用能为听障人士提供更智能、便捷的交流方式,将在未来得到更广泛的应用。In response to the needs of sign language recognition and translation,this paper first combines OpenCV and Mediapipe hands models to create a sign language dataset that includes various gestures,different angles,light and dark areas,simple and complex backgrounds,and other specific environments.The dataset consists of 3000 images in total.Secondly,in order to achieve more accurate and real-time sign language recognition and translation,the YOLO V5 model was selected for sign language recognition detection.On the test set,the average accuracy mAP(0.5)reached 99.1%.Then,combined with the Transformer model,isolated words in sign language are connected to generate sign language sentences.The application of these technologies can provide more intelligent and convenient communication methods for people with hearing impairment,and will be widely used in the future.

关 键 词：手语识别和翻译 Mediapipe hands YOLO V5 TRANSFORMER 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]