面向学前教育对话机器人的多模态情感识别实现关键技术  

Key Technologies of the Implementation of Multimodal Emotion Recognition for Preschool Education Dialogue Robots

作　　者：许萌 韩鹏[1] XU Meng;HAN Peng(Xianyang Vocational and Technical College,Xianyang Shaanxi 712000,China)

机构地区：[1]咸阳职业技术学院,陕西咸阳712000

出　　处：《自动化与仪器仪表》2023年第9期137-141,共5页Automation & Instrumentation

基　　金：陕西省职业技术教育学会2023年度教育教学改革研究课题项目《新时代高职院校劳动教育模式构建与实践创新研究》(2023SZX207);院级教改项目《新时代高职院校劳动教育模式构建与实践创新研究》(2023SZX212)。

摘　　要：为进一步提高学前教育对话机器人交互过程的准确性,结合多模态融合思想,提出一种基于面部表情情感和语音情感融合的识别技术。其中,为解决面部表情异常视频帧的问题,采用卷积神经网络对人脸进行检测,然后基于Gabor小波变换对人脸表情进行特征提取,最后通过残差网络对面部表情情感进行识别;为提高情感识别的准确性,协助学前教育机器人更好地理解儿童情感,在采用MFCC对连续语音特征进行提取后,通过残差网络对连续语音情感进行识别;利用多元线性回归算法对面部和语音情感识别结果进行融合。在AVEC2019数据集上的验证结果表明,表情情感识别和连续语音情感识别均具有较高识别精度;与传统的单一情感识别相比,多模态融合识别的一致性相关系数最高,达0.77。由此得出,将多模态情感识别的方法将有助于提高学前教育对话机器人交互过程中的情感识别水平,提高对话机器人的智能化。In order to further improve the accuracy of the interaction process of preschool education dialogue robots,a recognition technology based on facial expression emotion and speech emotion fusion is proposed by combining multimodal fusion ideas.Among them,to solve the problem of abnormal facial expression video frames,convolutional neural networks are used to detect faces,followed by feature extraction of facial expressions based on Gabor wavelet transform,and finally,residual networks are used to recognize facial expression emotions;To improve the accuracy of emotion recognition and assist preschool education robots in better understanding children's emotions,MFCC is used to extract continuous speech features,and residual networks are used to recognize continuous speech emotions;Use multiple linear regression algorithms to fuse facial and speech emotion recognition results.The validation results on the AVEC2019 dataset show that both facial expression emotion recognition and continuous speech emotion recognition have high recognition accuracy;Compared with traditional single emotion recognition,multimodal fusion recognition has the highest consistency correlation coefficient,reaching 0.77.From this,it can be concluded that the method of multimodal emotion recognition will help improve the level of emotion recognition in the interaction process of preschool education dialogue robots,and enhance the intelligence of dialogue robots.

关 键 词：学前教育 对话机器人 情感识别 多模态融合 卷积神经网络 

分 类 号：TP392[自动化与计算机技术—计算机应用技术]