基于Transformer-LSTM的闽南语唇语识别  

A Research on Minnan Dialect Lip-Language Recognition Based on Transformer-LSTM

作　　者：曾蔚[1,2,3] 罗仙仙 王鸿伟[1,2,3] ZENG Wei;LUO Xianxian;WANG Hongwei(School of Mathematics and Computer Science,Quanzhou Normal University,Quanzhou Fujian 362000,China;Fujian Provincial Key Laboratory of Data Intensive Computing,Quanzhou Fujian 362000,China;Key Laboratory of Intelligent Computing and Information Processing,Quanzhou Fujian 362000,China)

机构地区：[1]泉州师范学院数学与计算机科学学院,福建泉州362000 [2]福建省大数据管理新技术与知识工程重点实验室,福建泉州362000 [3]智能计算与信息处理福建省高等学校重点实验室,福建泉州362000

出　　处：《泉州师范学院学报》2024年第2期10-17,共8页Journal of Quanzhou Normal University

基　　金：福建省教育厅中青年教师教育科研项目(JAT200542)。

摘　　要：针对端到端句子级闽南语唇语识别的问题,提出一种基于Transformer和长短时记忆网络(LSTM)的编解码模型.编码器采用时空卷积神经网络及Transformer编码器用于提取唇读序列时空特征,解码器采用长短时记忆网络并结合交叉注意力机制用于文本序列预测.最后,在自建闽南语唇语数据集上进行实验.实验结果表明:模型能有效地提高唇语识别的准确率.An Encoder-Decoder Model based on Transformer and long short term memory(LSTM)was proposed for end-to-end sentence level Minnan dialect lip recognition.The encoder used a spatiotemporal convolutional neural network and Transformer encoder to extract spatiotemporal features of lip reading sequences.The decoder used a long-term and short-term memory network combined with cross attention mechanism for text sequence prediction.Finally,experiments were conducted on the self built Minnan dialect lip language dataset,and the experimental results showed that the model can effectively improve the accuracy of lip language recognition.

关 键 词：唇语识别 闽南语 TRANSFORMER 长短时记忆网络(LSTM) 用时空卷积神经网络 注意力机制 端到端模型 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]