CLIP-Flow:Decoding images encoded in CLIP space  

作　　者：Hao Ma Ming Li Jingyuan Yang Or Patashnik Dani Lischinski Daniel Cohen-Or Hui Huang 

机构地区：[1]Visual Computing Research Center,College of Computer Science and Software Engineering,Shenzhen University,Shenzhen 518060,China [2]Department of Computer Science,Tel Aviv University,Tel Aviv 6997801,Israel [3]School of Computer Science and Engineering,the Hebrew University of Jerusalem,Jerusalem 91904,Israel

出　　处：《Computational Visual Media》2024年第6期1157-1168,共12页计算可视媒体(英文版)

基　　金：supported in parts by the National Natural Science Foundation of China(62161146005,U21B2023);Shenzhen Science and Technology Program(KQTD20210811090044003,RCJC20200714114435012);Israel Science Foundation.

摘　　要：This study introduces CLIP-Flow,a novel network for generating images from a given image or text.To effectively utilize the rich semantics contained in both modalities,we designed a semantics-guided methodology for image-and text-to-image synthesis.In particular,we adopted Contrastive Language-Image Pretraining(CLIP)as an encoder to extract semantics and StyleGAN as a decoder to generate images from such information.Moreover,to bridge the embedding space of CLIP and latent space of StyleGAN,real NVP is employed and modified with activation normalization and invertible convolution.As the images and text in CLIP share the same representation space,text prompts can be fed directly into CLIP-Flow to achieve text-to-image synthesis.We conducted extensive experiments on several datasets to validate the effectiveness of the proposed image-to-image synthesis method.In addition,we tested on the public dataset Multi-Modal CelebA-HQ,for text-to-image synthesis.Experiments validated that our approach can generate high-quality text-matching images,and is comparable with state-of-the-art methods,both qualitatively and quantitatively.

关 键 词：image-to-image text-to-image contrastive language-image pretraining(CLIP) FLOW StyleGAN 

分 类 号：TP3[自动化与计算机技术—计算机科学与技术]