MILG:Realistic lip-sync video generation with audio-modulated image inpainting  

作　　者：Han Bao Xuhong Zhang Qinying Wang Kangming Liang Zonghui Wang Shouling Ji Wenzhi Chen 

机构地区：[1]School of Software Technology,Zhejiang University,Hangzhou,China [2]College of Computer Science,Zhejiang University,Hangzhou,China [3]College of Engineering,Zhejiang University,Hangzhou,China

出　　处：《Visual Informatics》2024年第3期71-81,共11页可视信息学（英文）

基　　金：partially funded by the National Key Research and Development Program of China(Grant No.2020AAA0140004).

摘　　要：Existing lip synchronization(lip-sync)methods generate accurately synchronized mouths and faces in a generated video.However,they still confront the problem of artifacts in regions of non-interest(RONI),e.g.,background and other parts of a face,which decreases the overall visual quality.To solve these problems,we innovatively introduce diverse image inpainting to lip-sync generation.We propose Modulated Inpainting Lip-sync GAN(MILG),an audio-constraint inpainting network to predict synchronous mouths.MILG utilizes prior knowledge of RONI and audio sequences to predict lip shape instead of image generation,which can keep the RONI consistent.Specifically,we integrate modulated spatially probabilistic diversity normalization(MSPD Norm)in our inpainting network,which helps the network generate fine-grained diverse mouth movements guided by the continuous audio features.Furthermore,to lower the training overhead,we modify the contrastive loss in lipsync to support small-batch-size and few-sample training.Extensive experiments demonstrate that our approach outperforms the existing state-of-the-art of image quality and authenticity while keeping lip-sync.

关 键 词：Lip-sync Image inpainting Face generation Modulated SPD normalization 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]