基于音频驱动的遮挡下人脸表情重建技术  

Technology of Audio-Driven Masked Facial Expression Reconstruction

作　　者：王航宇 李晓冬 李新德[3] WANG Hangyu;LI Xiaodong;LI Xinde(School of Cyber Science and Engineering,Southeast University,Nanjing 211189,China;Science and Technology on Information Systems Engineering Laboratory,Nanjing 210023,China;School of Automation,Southeast University,Nanjing 210096,China)

机构地区：[1]东南大学网络空间安全学院,南京211189 [2]信息系统工程重点实验室,南京210023 [3]东南大学自动化学院,南京210096

出　　处：《指挥信息系统与技术》2022年第6期89-94,共6页Command Information System and Technology

基　　金：信息系统工程重点实验室开放基金(05202003)资助项目。

摘　　要：网络舆情对政治生活秩序和社会稳定的影响与日俱增,还原网络视频中刻意遮挡的人脸,有助于网络舆情管控以及掌握网络舆情攻防战的主动权。结合生成对抗网络,提出了一种基于音频驱动的遮挡下人脸表情重建方法。首先,通过音频解耦器实现音频内容与情绪分离;然后,基于双阶段遮挡下人眼关键点检测网络对头部姿态进行估计,得到头部3维姿态编码;最后,基于生成对抗网络,对生成人脸的说话口型、面部表情和头部姿态进行综合控制,实现对说话人脸的生动还原。试验结果表明,该方法可准确有效地还原人脸表情和头部姿态。The influence of online public opinion on political life order and social stability is increasing day by day. Restoring the faces deliberately blocked in online videos is helpful to control online public opinion and seize the initiative in the battle against online public opinion. Combined with generative adversarial network, a method of audio-driven masked facial expression reconstruction is proposed. Firstly, the audio content and emotion are separated through audio decoupling device. Then, based on the two-stage masked human eye key-points detection network, the head pose is estimated to obtain the 3D head pose code. Finally, based on the generative adversarial network, the mouth shape, facial expression and head posture of the generated face are comprehensively controlled to achieve a vivid restoration of the speaker′s face. The experimental result shows that the method can effectively restore facial expressions and head poses.

关 键 词：网络舆情 人脸生成 深度学习 生成对抗网络 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]