融合扩散文本的图像描述算法  

An image caption algorithm based on fusion diffusion of text

作　　者：刘仲民 王学臻 胡文瑾[3] LIU Zhong-min;WANG Xue-zhen;HU Wen-jin(Key Laboratory of Gansu Advanced Control for Industrial Processes,College of Electrical and Information Engineering,Lanzhou University of Technology,Lanzhou 730050,China;Chongqing Chang'an Automobile Co.,Ltd.,Chongqing 400023,China;College of Mathematics and Computer Science,Northwest Minzu University,Lanzhou 730030,China)

机构地区：[1]兰州理工大学电气工程与信息工程学院,甘肃省工业过程先进控制重点实验室,兰州730050 [2]重庆长安汽车股份有限公司,重庆400023 [3]西北民族大学数学与计算机科学学院,兰州730030

出　　处：《兰州大学学报(自然科学版)》2025年第1期8-16,共9页Journal of Lanzhou University(Natural Sciences)

基　　金：国家自然科学基金项目(62061042);甘肃省工业过程先进控制重点实验室开发基金项目(2022KX10)。

摘　　要：针对图像描述任务中特征信息利用不充分、背景信息易被忽略、生成语句的可控性存在欠缺等问题,使用特征融合模块和基于Transformer编/解码器的文本解码增强网络,提出一种融合扩散文本的图像描述算法.输入图像由Vision Transformer骨干网络提取细粒度信息,并在解码器中将原始的BERT网络结构替换为新型的DeBERTa网络结构,提升了Transformer模型的解码效果.文本扩散方法对图像描述语句的生成过程进行监督,减少描述语句的无序化.用强化学习方式对模型进行微调,优化最终描述语句生成结果.将提出算法在MS-COCO数据集中进行定量评估和线上对比测试.结果表明,模型在BLEU-1、BLEU-4、METEOR、ROUGE-L、CIDEr、SPICE数据集上的性能分别提高2.1%、3.1%、1.7%、2%、3.2%、2.6%,且模型的微调过程更加稳定,对图像的语义表述更加准确有序.In order to solve the problem of insufficient feature utilization and easy neglect of image details in image caption,a feature reconstruction module and channel self-attention module were used to improve the image description method,and an adaptive feature reconstruction model based on Transformer designed to enhance visual features and text matching.The feature reconstruction module as the backbone network to extract grid features from images,implement feature extraction by convolution image checking at different scales,and adaptive channel attention module used to screen visual features for word weight redistribution.The model used cross entropy loss to train the function and reinforcement learning to fine-tune the model.Quantitative evaluation of and online comparison tests on MS-COCO datasets showed the superiority of this model.Experimental results indicated that the scores of BLEU-1,BLEU-4,METEOR,ROUGE-L,CIDEr and SPICE had increased by 2.1%,3.1%,1.7%,2%,3.2%,2.6%respectively and more detailed and accurate image descriptions could be generated compared to normal Transformer models.

关 键 词：图像描述 特征融合 Transformer模型 扩散模型 强化学习 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]