融合多重视觉特征与语义信息的图像描述生成  

Combining multiple visual features and semantic information for image captioning

作　　者：傅煦嘉 周家乐 王慧锋[1] 颜秉勇[1] FU Xu-jia;ZHOU Jia-le;WANG Hui-feng;YAN Bing-yong(College of Information Science and Technology,East China University of Science and Technology,Shanghai 200237,China)

机构地区：[1]华东理工大学信息科学与工程学院,上海200237

出　　处：《计算机工程与设计》2023年第4期1066-1072,共7页Computer Engineering and Design

基　　金：国家自然科学青年基金项目(61906068)。

摘　　要：针对图像描述模型中对语义信息考虑不足,循环神经网络收敛速度慢与精度低等问题,提出一种基于多注意力融合的深层图像描述模型。通过对图像中提取到的内容信息以及文本描述信息分配不同的权重,达到提升精度的效果,融合MOGRIFIER网络解决循环神经网络收敛速度缓慢的问题。使用改进模型与传统模型在数据集MSCOCO上进行对比实验,实验结果表明,该方法能够生成更加准确的描述,在BLEU与CIDEr等关键指标上有明显提升。A deep image caption model based on multi-attention fusion was proposed to solve the problems of insufficient conside-ration of semantic information,low convergence rate and low accuracy of recurrent neural network.The content information extracted from the image and the text description information was assigned with different weights to improve the accuracy.The problem of low convergence speed of the recurrent neural network was solved by integrating the MOGRIFIER network.The improved model was compared with the traditional model on the dataset MSCOCO.Experimental results show that the proposed method can generate more accurate description and significantly improve the key indicators,such as BLEU and CIDEr.

关 键 词：图像描述 多注意力融合 语义信息 深层图像描述模型 MOGRIFIER网络 收敛速度 精度 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]