融合rPPG和人脸三维法向量的非接触情绪感知技术  

Non-contact Emotion Recognition Technology Integrating rPPG and 3D Facial Normal Vectors

作　　者：王宇[1] 刘宇昂 赵梦洁 涂晓光 牛知艺 杨明[1] 刘建华 殷举航 朱新宇[1] 石臣鹏 章超[4] 张铖方 WANG Yu;LIU Yuang;ZHAO Mengjie;TU Xiaoguang;NIU Zhiyi;YANG MingLIU;Jianhua;YIN Juhang;ZHU Xinyu;SHI Chenpeng;ZHANG Chao;ZHANG Chengfang(College of Avionics and Electrical Engineering,Civil Aviation Flight University of China,Guanghan 2.Sichuan 618307,China;Province Engineering Technology Research Center of General Aircraft Civil Maintenance,Aviation Flight University of China,Guanghan 3.Affiliated 618307,China;Hospital of Chengdu University of Traditional Chinese Medicine,Chengdu 4.Sichuan 610032,China;Provincial Key Laboratory of Intelligent Policing,Luzhou 5.Department 646000,China;of Road Traffic Management,Sichuan Police College,Luzhou 646000,China)

机构地区：[1]中国民用航空飞行学院航空电子电气学院,四川广汉618307 [2]中国民用航空飞行学院四川省通用航空器维修工程技术研究中心,四川广汉618307 [3]成都中医药大学附属医院,成都610032 [4]智能警务四川省重点实验室,四川泸州646000 [5]四川警察学院道路交通管理系,四川泸州646000

出　　处：《电讯技术》2024年第10期1667-1676,共10页Telecommunication Engineering

基　　金：中国博士后科学基金(2022M722248);中央高校基本科研业务费(ZHMH2022-004,J2023-026);四川省通用航空器维修工程技术研究中心资助课题(GAMRC2023YB06);智能警务四川省重点实验室开放课题(ZNJW2024KFQN003)。

摘　　要：近年来,基于深度学习的情绪识别技术取得了显著进展。然而,现有方法主要集中在面部图像或视频上,忽略了其他模态信息,导致鲁棒性和稳定性不足。为了解决这一问题,提出了一种融合多模态信息的面部表情识别方法。首先将输入的人脸视频进行远程光电容积脉搏波(Remote Photo plethysmography,rPPG)信号和人脸三维法向量的提取,其次将这两种模态的信息输入其相应的情绪特征提取子网络,提取出对应的情绪特征向量。然后,将这两种模态提取出的情绪特征向量进行融合,生成一个丰富的特征向量,最后将其输入分类器进行情绪分类任务。通过这种多模态信息融合的方式,提高了面部表情识别的准确性和稳定性。对所提方法在不同数据集上进行了验证,实验结果表明,该方法在多样化面部表情识别中的表现优于当前先进的情感识别方法,具有更高的鲁棒性和稳定性。In recent years,emotion recognition technology based on deep learning has made significant progress.However,existing methods mainly focus on facial images or videos,neglecting other modal information,which leads to insufficient robustness and stability.To address this issue,a facial expression recognition method that integrating multimodal information is proposed.First,remote photo plethysmography(rPPG)signals and 3D facial normal vectors are extracted from the input facial video.Then,the information from these two modalities is fed into their respective emotion feature extraction sub-networks to extract the corresponding emotion feature vectors.After that,the emotion feature vectors extracted from these two modalities are fused to generate a rich feature vector,which is then fed into a classifier for emotion classification tasks.By integrating multimodal information,the accuracy and stability of facial expression recognition are improved.The proposed method has been validated on different datasets,and experimental results show that it outperforms current advanced emotion recognition methods in diverse facial expression recognition tasks,demonstrating higher robustness and stability.

关 键 词：非接触情绪感知 多模态表情识别 远程光电容积脉搏波描记法(rppG) 人脸三维法向量 

分 类 号：TP183[自动化与计算机技术—控制理论与控制工程]