多任务学习和对抗学习结合的自发与非自发表情识别  

Posed and spontaneous expression distinction through multi-task and adversarial learning

作　　者：郑壮强 姜其胜 王上飞[1] Zheng Zhuangqiang;Jiang Qisheng;Wang Shangfei(School of Computer Science and Technology,University of Science and Technology of China,Hefei 230027,China;School of Data Science,University of Science and Technology of China,Hefei 230027,China)

机构地区：[1]中国科学技术大学计算机科学与技术学院,合肥230027 [2]中国科学技术大学大数据学院,合肥230027

出　　处：《中国图象图形学报》2020年第11期2370-2379,共10页Journal of Image and Graphics

基　　金：安徽省重点研发计划项目(1804a09020038)。

摘　　要：目的如何提取与个体身份无关的面部特征以及建模面部行为的时空模式是自发与非自发表情识别的核心问题,然而现有的自发与非自发表情识别工作尚未同时兼顾两者。针对此,本文提出多任务学习和对抗学习结合的自发与非自发表情识别方法,通过多任务学习和对抗学习捕获面部行为的时空模式以及与学习身份无关的面部特征,实现有效的自发与非自发表情区分。方法所提方法包括4部分:特征提取器、多任务学习器、身份判别器以及多任务判别器。特征提取器用来获取与自发和非自发表情相关的特征;身份判别器用来监督特征提取器学习到的特征,与身份标签无关;多任务学习器预测表情高峰帧相对于初始帧之间的特征点偏移量以及表情类别,并试图迷惑多任务判别器;多任务判别器辨别输入是真实的还是预测的人脸特征点偏移量与表情类别。通过多任务学习器和多任务判别器之间的对抗学习,捕获面部行为的时空模式。通过特征提取器、多任务学习器和身份判别器的协同学习,学习与面部行为有关而与个体身份无关的面部特征。结果在MMI(M&M initiative)、NVIE(natural visible and infrared facial expression)和Bio Vid(biopotential and video)数据集上的实验结果表明本文方法可以学习出与个体身份相关性较低的特征,通过同时预测特征点偏移量和表情类别,有效捕获自发和非自发表情的时空模式,从而获得较好的自发与非自发表情识别效果。结论实验表明本文所提出的基于对抗学习的网络不仅可以有效学习个体无关但表情相关的面部中特征,而且还可以捕捉面部行为中的空间模式,而这些信息可以很好地改善自发与非自发表情识别。Objective Posed and spontaneous expression distinction is a major problem in the field of facial expression analysis.Posed expressions are deliberately performed to confuse or cheat others,while spontaneous expressions occur naturally.The difference between the posed and spontaneous delivery of the same expression by a person is little due to the subjective fraud of posed expressions.At the same time,posed and spontaneous expression distinction suffers from the problem of high intraclass differences caused by individual differences.These limitations bring difficulties in posed and spontaneous expression distinction.However,behavioral studies have shown that significant differences exist between posed and spontaneous expressions in spatial patterns.For example,compared with spontaneous smiles,the contraction of zygomatic muscles is more likely to be asymmetric in posed smiles.Moreover,constricted orbicularis oculi muscle is presented in spontaneous smiles but absent in posed smiles.Such inherent spatial patterns in posed and spontaneous expressions can be utilized to facilitate posed and spontaneous expression distinction.Therefore,modeling spatial patterns inherent in facial behavior and extracting subject-independent facial features are important for posed and spontaneous expression distinction.Previous works typically focused on spatial pattern modeling in the facial behavior.Researchers commonly use landmarks to describe motion patterns of facial muscles approximately and capture spatial patterns inherent in facial behavior based on landmark information due to the difficulty in obtaining motion patterns of facial muscles.According to the difference in modeling spatial patterns inherent in facial behavior,studies on posed and spontaneous expression distinction can be categorized into two approaches,namely,feature-and probabilistic graphical model(PGM)-based methods.Feature-based methods implicitly capture spatial patterns using handcrafted low-level or deep features extracted by deep convolution networks.Howeve

关 键 词：自发与非自发表情识别 对抗学习 多任务学习 面部行为的时空模式 个体身份无关的面部特征 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]