DiffMat:Latent diffusion models for image-guided material generation  

作　　者：Liang Yuan Dingkun Yan Suguru Saito Issei Fujishiro 

机构地区：[1]Graduate School of Science and Technology,Keio University,Kanagawa,Japan [2]School of Computing,Tokyo Institute of Technology,Tokyo,Japan [3]Department of Information and Computer Science,Keio University,Kanagawa,Japan

出　　处：《Visual Informatics》2024年第1期6-14,共9页可视信息学（英文）

基　　金：Grant-in-Aid for Scientific Research(A)JP21H04916 and the Research Grant of Keio Leading-edge Laboratory of Science and Technology,Japan.

摘　　要：Creating realistic materials is essential in the construction of immersive virtual environments.While existing techniques for material capture and conditional generation rely on flash-lit photos,they often produce artifacts when the illumination mismatches the training data.In this study,we introduce DiffMat,a novel diffusion model that integrates the CLIP image encoder and a multi-layer,crossattention denoising backbone to generate latent materials from images under various illuminations.Using a pre-trained StyleGAN-based material generator,our method converts these latent materials into high-resolution SVBRDF textures,a process that enables a seamless fit into the standard physically based rendering pipeline,reducing the requirements for vast computational resources and expansive datasets.DiffMat surpasses existing generative methods in terms of material quality and variety,and shows adaptability to a broader spectrum of lighting conditions in reference images.

关 键 词：SVBRDF Diffusion model Generative model Appearance modeling 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]