基于扩散生成对抗网络的文本生成图像模型研究  

Text-to-image Generation Model Based on Diffusion Wasserstein Generative Adversarial Networks

作　　者：赵宏[1] 李文改 ZHAO Hong;LI Wengai(School of Computing and Communication,Lanzhou University of Technology,Lanzhou 730050,China)

机构地区：[1]兰州理工大学计算机与通信学院,兰州730050

出　　处：《电子与信息学报》2023年第12期4371-4381,共11页Journal of Electronics & Information Technology

基　　金：国家自然科学基金(62166025);甘肃省重点研发计划(21YF5GA073)。

摘　　要：文本生成图像是一项结合计算机视觉(CV)和自然语言处理(NLP)领域的综合性任务。以生成对抗网络(GANs)为基础的方法在文本生成图像方面取得了显著进展,但GANs方法的模型存在训练不稳定的问题。为解决这一问题,该文提出一种基于扩散Wasserstein生成对抗网络(WGAN)的文本生成图像模型(D-WGAN)。在DWGAN中,利用向判别器中输入扩散过程中随机采样的实例噪声,在实现模型稳定训练的同时,生成高质量和多样性的图像。考虑到扩散过程的采样成本较高,引入一种随机微分的方法,以简化采样过程。为了进一步对齐文本与图像的信息,提出使用基于对比学习的语言-图像预训练模型(CLIP)获得文本与图像信息之间的跨模态映射关系,从而提升文本和图像的一致性。在MSCOCO,CUB-200数据集上的实验结果表明,D-WGAN在实现稳定训练的同时,与当前最好的方法相比,FID分数分别降低了16.43%和1.97%,IS分数分别提升了3.38%和30.95%,说明D-WGAN生成的图像质量更高,更具有实用价值。Text-to-image generation is a comprehensive task that combines the fields of Computer Vision(CV)and Natural Language Processing(NLP).Research on the methods of text to image based on Generative Adversarial Networks(GANs)continues to grow in popularity and have made some progress,but the methods of GANs model suffer from training instability.To address this problem,a text-to-image generation model based on Diffusion Wasserstein Generative Adversarial Networks(D-WGAN)is proposed,which generates high quality and diverse images and enables stable training process by feeding randomly sampled instance noise from the diffusion process into the discriminator.Considering the high cost of sampling the diffusion process,a stochastic differentiation method is introduced to simplify the sampling process.In order to align further the information of text and image,Contrastive Language-Image Pre-training(CLIP)model is introduced to obtain the cross-modal mapping relationship between text and image information,so as to improve the consistency of text and image.Experimental results on the MSCOCO and CUB-200 datasets show that D-WGAN achieves stable training while reducing Fréchet Inception Distance(FID)scores by 16.43%and 1.97%,respectively,and improving Inception Score(IS)scores by 3.38%and 30.95%,respectively.These results indicate that D-WGAN can generate higher quality images and has more practical value.

关 键 词：文本生成图像 生成对抗网络 扩散过程 对比学习的语言-图像预训练模型 语义匹配 

分 类 号：TN911.73[电子电信—通信与信息系统] TP183[电子电信—信息与通信工程]