HLR-Net: A Hybrid Lip-Reading Model Based on Deep Convolutional Neural Networks  

作　　者：Amany M.Sarhan Nada M.Elshennawy Dina M.Ibrahim 

机构地区：[1]Department of Computers and Control Engineering,Faculty of Engineering,Tanta University,Tanta,37133,Egypt [2]Department of Information Technology,College of Computer,Qassim University,Buraydah,51452,Saudi Arabia

出　　处：《Computers, Materials & Continua》2021年第8期1531-1549,共19页计算机、材料和连续体（英文）

摘　　要：Lip reading is typically regarded as visually interpreting the speaker’s lip movements during the speaking.This is a task of decoding the text from the speaker’s mouth movement.This paper proposes a lip-reading model that helps deaf people and persons with hearing problems to understand a speaker by capturing a video of the speaker and inputting it into the proposed model to obtain the corresponding subtitles.Using deep learning technologies makes it easier for users to extract a large number of different features,which can then be converted to probabilities of letters to obtain accurate results.Recently proposed methods for lip reading are based on sequence-to-sequence architectures that are designed for natural machine translation and audio speech recognition.However,in this paper,a deep convolutional neural network model called the hybrid lip-reading(HLR-Net)model is developed for lip reading from a video.The proposed model includes three stages,namely,preprocessing,encoder,and decoder stages,which produce the output subtitle.The inception,gradient,and bidirectional GRU layers are used to build the encoder,and the attention,fully-connected,activation function layers are used to build the decoder,which performs the connectionist temporal classification(CTC).In comparison with the three recent models,namely,the LipNet model,the lip-reading model with cascaded attention(LCANet),and attention-CTC(A-ACA)model,on the GRID corpus dataset,the proposed HLR-Net model can achieve significant improvements,achieving the CER of 4.9%,WER of 9.7%,and Bleu score of 92%in the case of unseen speakers,and the CER of 1.4%,WER of 3.3%,and Bleu score of 99%in the case of overlapped speakers.

关 键 词：LIP-READING visual speech recognition deep neural network connectionist temporal classification 

分 类 号：H31[语言文字—英语]