A Survey of Multimodal Controllable Diffusion Models  

作　　者：江锐 郑光聪 李藤 杨天瑞 王井东 李玺 Rui Jiang;Guang-Cong Zheng;Teng Li;Tian-Rui Yang;Jing-Dong Wang;Xi Li(College of Computer Science and Technology,Zhejiang University,Hangzhou 310007,China;Department of Mathematics,Nanjing University,Nanjing 210023,China;Baidu Visual Technology Department,Baidu Inc.,Beijing 100085,China;IEEE)

机构地区：[1]College of Computer Science and Technology,Zhejiang University,Hangzhou,310007,China [2]Department of Mathematics,Nanjing University,Nanjing 210023,China [3]Baidu Visual Technology Department,Baidu Inc.,Beijing 100085,China [4]IEEE

出　　处：《Journal of Computer Science & Technology》2024年第3期509-541,共33页计算机科学技术学报（英文版）

基　　金：supported in part by the National Science Foundation for Distinguished Young Scholars of China under Grant No.62225605;the National Natural Science Foundation of China under Grant No.U20A20222;the Zhejiang Provincial Natural ScienceFoundation of China under Grant No.LD24F020016;the Ng Teng Fong Charitable Foundation in the form of ZJU-SUTDIDEA under Grant No.188170-11102。

摘　　要：Diffusion models have recently emerged as powerful generative models,producing high-fidelity samples across domains.Despite this,they have two key challenges,including improving the time-consuming iterative generation process and controlling and steering the generation process.Existing surveys provide broad overviews of diffusion model advancements.However,they lack comprehensive coverage specifically centered on techniques for controllable generation.This survey seeks to address this gap by providing a comprehensive and coherent review on controllable generation in diffusion models.We provide a detailed taxonomy defining controlled generation for diffusion models.Controllable generation is categorized based on the formulation,methodologies,and evaluation metrics.By enumerating the range of methods researchers have developed for enhanced control,we aim to establish controllable diffusion generation as a distinct subfield warranting dedicated focus.With this survey,we contextualize recent results,provide the dedicated treatment of controllable diffusion model generation,and outline limitations and future directions.To demonstrate applicability,we highlight controllable diffusion techniques for major computer vision tasks application.By consolidating methods and applications for controllable diffusion models,we hope to catalyze further innovations in reliable and scalable controllable generation.

关 键 词：diffusion model controllable generation APPLICATION PERSONALIZATION 

分 类 号：TP39[自动化与计算机技术—计算机应用技术]