头姿鲁棒的双一致性约束半监督表情识别  

Semi-supervised facial expression recognition robust to head pose empowered by dual consistency constraints

作　　者：王宇键 何军 张建勋[1] 孙仁浩 刘学亮[3] Wang Yujian;He Jun;Zhang Jianxun;Sun Renhao;Liu Xueliang(School of Computer Science and Engineering,Chongqing University of Technology,Chongqing 400054,China;Institute of Dataspace,Hefei 230088,China;School of Computer Science and Information Engineering,Hefei University of Technology,Hefei 230601,China)

机构地区：[1]重庆理工大学计算机科学与工程学院,重庆400054 [2]数据空间研究院,合肥230088 [3]合肥工业大学计算机与信息学院,合肥230601

出　　处：《中国图象图形学报》2025年第2期435-450,共16页Journal of Image and Graphics

基　　金：国家自然科学基金项目(61971078);重庆市教委科技重大项目(KJZD-M202301901)。

摘　　要：目的现有表情识别方法聚焦提升模型的整体识别准确率,对方法的头部姿态鲁棒性研究不充分。在实际应用中,人的头部姿态往往变化多样,影响表情识别效果,因此研究头部姿态对表情识别的影响,并提升模型在该方面的鲁棒性显得尤为重要。为此,在深入分析头部姿态对表情识别影响的基础上,提出一种能够基于无标签非正脸表情数据提升模型头部姿态鲁棒性的半监督表情识别方法。方法首先按头部姿态对典型表情识别数据集AffectNet重新划分,构建了AffectNet-Yaw数据集,支持在不同角度上进行模型精度测试,提升了模型对比公平性。其次,提出一种基于双一致性约束的半监督表情识别方法(dual-consistency semi-supervised learning for facial expression recognition,DCSSL),利用空间一致性模块对翻转前后人脸图像的类别激活一致性进行空间约束,使模型训练时更关注面部表情关键区域特征;利用语义一致性模块通过非对称数据增强和自学式学习方法不断地筛选高质量非正脸数据用于模型优化。在无需对非正脸表情数据人工标注的情况下,方法直接从有标签正脸数据和无标签非正脸数据中学习。最后,联合优化了交叉熵损失、空间一致性约束损失和语义一致性约束损失函数,以确保有监督学习和半监督学习之间的平衡。结果实验结果表明,头部姿态对自然场景表情识别有显著影响;提出AffectNet-Yaw具有更均衡的头部姿态分布,有效促进了对这种影响的全面评估;DCSSL方法结合空间一致性和语义一致性约束充分利用无标签非正脸表情数据,显著提高了模型在头部姿态变化下的鲁棒性,较MA-NET(multi-scale and local attention network)和EfficientFace全监督方法,平均表情识别精度分别提升了5.40%和17.01%。结论本文提出的双一致性半监督方法能充分利用正脸和非正脸数据,显著提升了模型在头部姿态变�Objective The field of facial expression recognition(FER)has long been a vibrant area of research,with a focus on improving the accuracy of identifying expressions across a wide range of faces.However,despite these advance⁃ments,a crucial aspect that has not been adequately explored is the robustness of FER models to changes in head pose.In real-world applications,where faces are captured at various angles and poses,existing methods often struggle to accurately recognize expressions in faces with considerable pose variations.This limitation has created an urgent need to understand the extent to which head pose affects FER models and to develop robust models that can handle diverse poses effectively.First,the impact of head pose on FER was analyzed.Rigorous experimentation has provided strong evidence that existing FER approaches are indeed vulnerable when faced with faces exhibiting large head poses.This vulnerability not only limits the practical applicability of these methods but also highlights the critical need for research focused on enhancing the pose robustness of FER models.This challenge is addressed by introducing a semi-supervised framework that leverages unla⁃beled nonfrontal facial expression samples to increase the pose robustness of FER models.This framework aims to over⁃come the limitations of existing methods by exploring unlabeled data to supplement labeled frontal face data,allowing the model to learn representations that are invariant to head pose variations.Incorporating unlabeled data expands the model’s exposure to a wider range of poses,ultimately enhancing robustness and accuracy in FER.This study highlights the impor⁃tance of pose robustness in FER and proposes a semi-supervised framework to address this critical limitation.Rigorous experimentation and analysis provide insights into the impact of head pose on FER,and a robust model to accurately recog⁃nize facial expressions across diverse poses is developed.This approach paves the way for more practical and reliable FER syst

关 键 词：表情识别(FER) 头部姿态 双一致性约束 半监督学习 AffectNet 图像识别 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]