基于加权融合的语音表情多模态情感识别方法  

Speech and Expression Multi-Modal Emotion Recognition Method Using Weighted Fusion

作　　者：焦爽 陈光辉 JIAO Shuang;CHEN Guang-hui(Central South Electric Power Test Resear ch Institute,China Datang Group Science and Technology Research Institute Co.,Ltd.,Zhengzhou Henan 450000,China;School of Information Science and Enginee ring,Southeast University,Nanjing Jiangsu 210096,China)

机构地区：[1]中国大唐集团科学技术研究院有限公司中南电力试验研究院,河南郑州450000 [2]东南大学信息科学与工程学院,江苏南京210096

出　　处：《计算机仿真》2024年第7期417-422,428,共7页Computer Simulation

基　　金：国家自然科学基金(U1504622)。

摘　　要：当前的多模态融合方法未充分利用语音和表情模态之间的互补性,导致多模态情感识别的识别率较低。为解决上述问题,提出了一种基于加权融合的语音表情多模态情感识别方法。方法首先利用语音活动检测(Voice Activation Detection, VAD)算法提取语音关键帧;然后,利用信息熵建模情感产生是一个连续的过程,并提取表情关键帧;其次,为充分利用语音和表情模态之间的互补性,采用语音和表情关键帧对齐技术计算语音和表情权重。这些权重被输进特征融合层进行加权融合,有效地提高多模态情感识别的识别率;最后,在RML、eNTERFACE05和BAUM-1s数据集上的实验结果表明,该上述方法的识别率高于其它基准方法。Current multi-modal fusion methods do not fully use the complementarity between speech and expression modalities,which results in low recognition rates for multi-modal emotion recognition.Thus,to solve this problem,this paper proposes a speech and expression multi-modal emotion recognition method based on weighted fusion.The method first uses the voice activation de tection(VAD)algorithm to extract speech key frames.Then,the information entropy is used to model that the genera tion of emotion is a continuous process,and the expression key frames are extracted.In addition,in order to fully use the complementarity between speech and expre ssion modalities,the speech and expression key frame alignment techniques are utilized to calculate speech and expres sion weights.These weights are input into the feature fusion layer for weighted fusion,which effectively improves the recognition rate of speech and expression multi-modal emotion recognition.Finally,the experimental results on the RML,eNTERFACE05 and BAUM-1s datasets show that the recognition rate of this method is higher than other bench mark methods.

关 键 词：情感识别 语音 表情 加权融合 

分 类 号：TP391.9[自动化与计算机技术—计算机应用技术]