A Multi-Modal Deep Learning Approach for Emotion Recognition  

作　　者：H.M.Shahzad Sohail Masood Bhatti Arfan Jaffar Muhammad Rashid 

机构地区：[1]The Superior University,Lahore,Pakistan [2]National University of Technology,Islamabad,Pakistan [3]Intelligent Data Visual Computing Research(IDVCR),Lahore,Pakistan

出　　处：《Intelligent Automation & Soft Computing》2023年第5期1561-1570,共10页智能自动化与软计算（英文）

基　　金：sponsored by The Superior University,Lahore.

摘　　要：In recent years,research on facial expression recognition(FER)under mask is trending.Wearing a mask for protection from Covid 19 has become a compulsion and it hides the facial expressions that is why FER under the mask is a difficult task.The prevailing unimodal techniques for facial recognition are not up to the mark in terms of good results for the masked face,however,a multi-modal technique can be employed to generate better results.We proposed a multi-modal methodology based on deep learning for facial recognition under a masked face using facial and vocal expressions.The multimodal has been trained on a facial and vocal dataset.We have used two standard datasets,M-LFW for the masked dataset and CREMA-D and TESS dataset for vocal expressions.The vocal expressions are in the form of audio while the faces data is in image form that is why the data is heterogenous.In order to make the data homogeneous,the voice data is converted into images by taking spectrogram.A spectrogram embeds important features of the voice and it converts the audio format into the images.Later,the dataset is passed to the multimodal for training.neural network and the experimental results demonstrate that the proposed multimodal algorithm outsets unimodal methods and other state-of-the-art deep neural network models.

关 键 词：Deep learning facial expression recognition multi-model neural network speech emotion recognition SPECTROGRAM covid-19 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]