基于深度学习的中国手语翻译  

Automated Chinese Sign Language Interpretation Using Deep Learning

作　　者：袁甜甜[1] 胡彬[1] 杨学[1] 赵伟[1] YUAN Tiantian;HU Bin;YANG Xue;ZHAO Wei(Technical College for the Deaf,Tianjin University of Technology,Tianjin 300384,China)

机构地区：[1]天津理工大学聋人工学院,天津300384

出　　处：《电视技术》2019年第2期52-55,80,共5页Video Engineering

基　　金：天津市工业企业发展专项资金项目(201707123);天津市软件产业发展专项资金项目(201606213)

摘　　要：本文提出了一种"sequence-to-sequence"的方法来解决中国手语的翻译问题。基本模型包括用于视频特征提取的预训练CNN和用于脱机机器翻译的两层LSTM。为了提高性能,我们还引入一些开源工具用于捕获身体姿态并注释来作为额外的特征。同时,使用Kinect2.0采集的大量样本作为训练数据,并提出一种面向手语者的时空注意力模型,以提高翻译的准确性。为了惠及其他研究人员和聋人,并帮助推广中国通用手语,我们已建立了一个较为庞大的连续中文手语语料库,并准备在全球范围内进行部分数据的共享。We propose a "sequence-to-sequence" approach to solve Chinese sign language interpretation problem. The baseline model will consist of a pre-trained CNN for video feature extraction, and a two-layer LSTM for offline machine translation. To improve the performance, we will also introduce body pose annotations captured by open-source tool as extra features. At this time, there are thousands of training samples that are captured by Kinect2.0. We also propose a signer-oriented spatial-temporal attention mechanism to boost the translation accuracy. To benefit other researchers and deaf people, and to help popularize the general Chinese sign language, we have built a huge Chinese sign language dataset and prepare to release some of them for researchers all over the world.

关 键 词：手语识别 神经机器翻译 CNN 时空注意力模型 语料库 

分 类 号：TP387[自动化与计算机技术—计算机系统结构]