多维度注意力和语义再生的文本生成图像模型  

Text-to-image Model by Multidimensional Attention and Semantic Regeneration

作　　者：庄兴旺 丁岳伟[1] ZHUANG Xing-wang;DING Yue-wei(School of Option-electrical and Computer Engineering,University of Shanghai for Science and Technology,Shanghai 200093,China)

机构地区：[1]上海理工大学光电信息与计算机工程学院,上海200093

出　　处：《计算机技术与发展》2020年第12期27-33,共7页Computer Technology and Development

基　　金：上海重点科技攻关项目(16DZ1203603);上海市工程中心建设项目(GCZX14014)。

摘　　要：文本生成图像是结合计算机视觉和自然语言处理两个领域的综合性任务,从给定的文本描述生成图像有两个目标:视觉真实性和语义一致性。虽然在使用生成对抗网络(GAN)生成高质量和视觉逼真的图像方面取得了显著进展,但确保文本描述和视觉内容之间的语义一致性仍然是非常具有挑战性的。目前的方法由于文本和图像形式的多样性,仅在单词级别使用注意力并不能确保全局语义的一致性。因此,在MirrorGAN的基础上提出了一种改进的多维度的注意力协同模块(MCAM)和语义文本再生模块(STRM)来解决这些问题。MCAM使用了更为先进的BERT模型来进行文本处理,STRM用于从生成的图像中重新生成文本描述,该图像在语义上与给定的文本描述对齐,使生成的图像更加贴合语义。最后,形成了基于多维度注意力以及语义文本再生的生成对抗网络模型(MirrorGAN++)。通过对两个公共基准数据集的深入实验,证明了MirrorGAN++优于其他方法。Text-to-image is a comprehensive task combining computer vision and natural language processing.Generating an image from a given text description has two goals:visual realism and semantic consistency.Although significant progress has been made in generating high-quality and visually realistic images using generative adversarial networks,guaranteeing semantic consistency between the text description and visual content remains challenging.The current approaches only using word-level attention cannot ensure global semantic consistency due to the diverse nature of both the text and image modalities.Therefore,we propose an improved multidimensional collaborative attentive module(MCAM)and semantic text regeneration module(STRM)based on MirrorGAN to solve these problems.MCAM uses a more advanced BERT model for text processing,and STRM is used to regenerate the text description from the generated image.The image is semantically aligned with the given text description,making the generated image more suitable for the semantics.Finally,a generative adversarial network model based on multi-dimensional attention and semantic text regeneration(MirrorGAN++)is formed.Thorough experiments on two public benchmark datasets demonstrate the superiority of MirrorGAN++over other representative state-of-the-art methods.

关 键 词：文本生成图像 生成对抗网络 语义一致 注意力 语义文本再生 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]