Multi3D:3D-aware multimodal image synthesis  

作　　者：Wenyang Zhou Lu Yuan Taijiang Mu 

机构地区：[1]BNRist,Tsinghua University,Beijing 100084,China [2]Computer Science Department,Stanford University,California 94305,USA

出　　处：《Computational Visual Media》2024年第6期1205-1217,共13页计算可视媒体(英文版)

基　　金：supported by the National Science and Technology Major Project(Grant No.2021ZD0112902);the National Natural Science Foundation of China(Project No.62220106003);a Research Grant from Beijing Higher Institution Engineering Research Center,and Tsinghua–Tencent Joint Laboratory for Internet Innovation Technology.

摘　　要：3D-aware image synthesis has attained high quality and robust 3D consistency.Existing 3D controllable generative models are designed to synthesize 3D-aware images through a single modality,such as 2D segmentation or sketches,but lack the ability to finely control generated content,such as texture and age.In pursuit of enhancing user-guided controllability,we propose Multi3D,a 3D-aware controllable image synthesis model that supports multi-modal input.Our model can govern the geometry of the generated image using a 2D label map,such as a segmentation or sketch map,while concurrently regulating the appearance of the generated image through a textual description.To demonstrate the effectiveness of our method,we have conducted experiments on multiple datasets,including CelebAMask-HQ,AFHQ-cat,and shapenet-car.Qualitative and quantitative evaluations show that our method outperforms existing state-of-the-art methods.

关 键 词：generate adversarial networks(GANs) neural radiation field(NeRF) 3D-aware image synthesis controllable generation 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]