Pyramid-VAE-GAN:Transferring hierarchical latent variables for image inpainting  

作　　者：Huiyuan Tian Li Zhang Shijian Li Min Yao Gang Pan 

机构地区：[1]College of Computer Science and Technology,Zhejiang University,Hangzhou 310027,China [2]Advanced Technology Research Institute,Zhejiang University,Hangzhou 310027,China

出　　处：《Computational Visual Media》2023年第4期827-841,共15页计算可视媒体（英文版）

基　　金：The authors gratefully acknowledge the financial support of the National Natural Science Foundation of China(Grant No.61925603).

摘　　要：Significant progress has been made in image inpainting methods in recent years.However,they are incapable of producing inpainting results with reasonable structures,rich detail,and sharpness at the same time.In this paper,we propose the Pyramid-VAE-GAN network for image inpainting to address this limitation.Our network is built on a variational autoencoder(VAE)backbone that encodes high-level latent variables to represent complicated high-dimensional prior distributions of images.The prior assists in reconstructing reasonable structures when inpainting.We also adopt a pyramid structure in our model to maintain rich detail in low-level latent variables.To avoid the usual incompatibility of requiring both reasonable structures and rich detail,we propose a novel cross-layer latent variable transfer module.This transfers information about long-range structures contained in high-level latent variables to low-level latent variables representing more detailed information.We further use adversarial training to select the most reasonable results and to improve the sharpness of the images.Extensive experimental results on multiple datasets demonstrate the superiority of our method.Our code is available at https://github.com/thy960112/Pyramid-VAE-GAN.

关 键 词：image inpainting variational autoencoder(VAE) latent variable transfer(LTN) pyramid structure generative model 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]