An approach based on deep learning for Indian sign language translation  

作　　者：Kinjal Bhargavkumar Mistree Devendra Thakor Brijesh Bhatt 

机构地区：[1]Chhotubhai Gopalbhai Patel Institute of Technology,Uka Tarsadia University,Bardoli,India [2]Dharmsinh Desai Institute of Technology,Dharmsinh Desai University,Nadiad,India

出　　处：《International Journal of Intelligent Computing and Cybernetics》2023年第3期397-419,共23页智能计算与控制论国际期刊（英文）

摘　　要：Purpose–According to the Indian Sign Language Research and Training Centre(ISLRTC),India has approximately 300 certified human interpreters to help people with hearing loss.This paper aims to address the issue of Indian Sign Language(ISL)sentence recognition and translation into semantically equivalent English text in a signer-independent mode.Design/methodology/approach–This study presents an approach that translates ISL sentences into English text using the MobileNetV2 model and Neural Machine Translation(NMT).The authors have created an ISL corpus from the Brown corpus using ISL grammar rules to perform machine translation.The authors’approach converts ISL videos of the newly created dataset into ISL gloss sequences using the MobileNetV2 model and the recognized ISL gloss sequence is then fed to a machine translation module that generates an English sentence for each ISL sentence.Findings–As per the experimental results,pretrained MobileNetV2 model was proven the best-suited model for the recognition of ISL sentences and NMT provided better results than Statistical Machine Translation(SMT)to convert ISL text into English text.The automatic and human evaluation of the proposed approach yielded accuracies of 83.3 and 86.1%,respectively.Research limitations/implications–It can be seen that the neural machine translation systems produced translations with repetitions of other translated words,strange translations when the total number of words per sentence is increased and one or more unexpected terms that had no relation to the source text on occasion.The most common type of error is the mistranslation of places,numbers and dates.Although this has little effect on the overall structure of the translated sentence,it indicates that the embedding learned for these few words could be improved.Originality/value–Sign language recognition and translation is a crucial step toward improving communication between the deaf and the rest of society.Because of the shortage of human interpreters,an alternative appro

关 键 词：Indian sign language Neural machine translation ISL corpus Pretrained models Sign language recognition Sign language translation Paper type Research paper 

分 类 号：H315.9[语言文字—英语]