MusicFace: Music-driven expressive singing face synthesis  

作　　者：Pengfei Liu Wenjin Deng Hengda Li Jintai Wang Yinglin Zheng Yiwei Ding Xiaohu Guo Ming Zeng 

机构地区：[1]School of Informatics,Xiamen University,Xiamen,361000,China [2]Department of Computer Science,The University of Texas at Dallas,Richardson,Texas,75080-3021,USA

出　　处：《Computational Visual Media》2024年第1期119-136,共18页计算可视媒体（英文版）

基　　金：This work was supported in part by grants from the National Key R&D Program of China(2021YFC3300403);National Natural Science Foundation of China(62072382);Yango Charitable Foundation,and the National Science Foundation(OAC-2007661).

摘　　要：It remains an interesting and challenging problem to synthesize a vivid and realistic singing face driven by music. In this paper, we present a method for this task with natural motions for the lips, facial expression, head pose, and eyes. Due to the coupling of mixed information for the human voice and backing music in common music audio signals, we design a decouple-and-fuse strategy to tackle the challenge. We first decompose the input music audio into a human voice stream and a backing music stream. Due to the implicit and complicated correlation between the two-stream input signals and the dynamics of the facial expressions, head motions, and eye states, we model their relationship with an attention scheme, where the effects of the two streams are fused seamlessly. Furthermore, to improve the expressivenes of the generated results, we decompose head movement generation in terms of speed and direction, and decompose eye state generation into short-term blinking and long-term eye closing, modeling them separately. We have also built a novel dataset, SingingFace, to support training and evaluation of models for this task, including future work on this topic. Extensive experiments and a user study show that our proposed method is capable of synthesizing vivid singing faces, qualitatively and quantitatively better than the prior state-of-the-art.

关 键 词：face synthesis SINGING MUSIC generative adversarial network 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]