Fine-Grained Cross-Modal Fusion Based Refinement for Text-to-Image Synthesis  

作　　者：SUN Haoran WANG Yang LIU Haipeng QIAN Biao 

机构地区：[1]Key Laboratory of Knowledge Engineering with Big Data,Ministry of Education,Hefei University of Technology,Hefei 230000,China [2]Department of Computer Science and Information Engineering,Hefei University of Technology,Hefei 230000,China

出　　处：《Chinese Journal of Electronics》2023年第6期1329-1340,共12页电子学报（英文版）

基　　金：supported by the National Natural Science Foundation of China(U21A20470,62172136,U1936217);the Key Research and Technology Development Projects of Anhui Province(202004 a5020043).

摘　　要：Text-to-image synthesis refers to generating visual-realistic and semantically consistent images from given textual descriptions.Previous approaches generate an initial low-resolution image and then refine it to be high-resolution.Despite the remarkable progress,these methods are limited in fully utilizing the given texts and could generate text-mismatched images,especially when the text description is complex.We propose a novel finegrained text-image fusion based generative adversarial networks(FF-GAN),which consists of two modules:Finegrained text-image fusion block(FF-Block)and global semantic refinement(GSR).The proposed FF-Block integrates an attention block and several convolution layers to effectively fuse the fine-grained word-context features into the corresponding visual features,in which the text information is fully used to refine the initial image with more details.And the GSR is proposed to improve the global semantic consistency between linguistic and visual features during the refinement process.Extensive experiments on CUB-200 and COCO datasets demonstrate the superiority of FF-GAN over other state-of-the-art approaches in generating images with semantic consistency to the given texts.

关 键 词：Text-to-image synthesis Text-image fusion Generative adversarial network 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]