融合多时间维度视觉与语义信息的图像描述方法  

Image Captioning Method for Fusing Multi-temporal Dimensional Visual and Semantic Information

作　　者：陈善学[1] 王程 CHEN Shanxue;WANG Cheng(School of Communication and Information Engineering,Chongqing University of Posts and Telecommunications,Chongqing 400065,China)

机构地区：[1]重庆邮电大学通信与信息工程学院,重庆400065

出　　处：《数据采集与处理》2024年第4期922-932,共11页Journal of Data Acquisition and Processing

摘　　要：传统的图像描述方法仅使用当前时刻的视觉信息和语义信息来生成预测词,而没有考虑过去时刻的视觉信息和语义信息,从而导致模型输出的信息在时间维度上比较单一,因此生成的描述语句在准确性上有所欠缺。针对此问题,提出一种融合多时间维度视觉与语义信息的图像描述方法,有效地融合了过去时刻的视觉信息和语义信息,并设计一种门控机制动态地对两种信息进行选择利用。在MSCOCO数据集上进行实验验证,结果表明该方法能够更准确地生成描述语句,和当前最主流的图像描述方法进行对比,性能在各项评价指标上都得到了可观的提升。Traditional image captioning methods use only the visual and semantic information of the current moment to generate prediction words without considering the visual and semantic information of the past moments,which leads to the output of the model to be relatively homogeneous in terms of temporal dimension.As a result,the generated captioning is lacking in terms of accuracy.To address this problem,an image captioning method that fuses multi-temporal dimensional visual and semantic information is proposed,which effectively fuses visual and semantic information of past moments and designs a gating mechanism to dynamically select both kinds of information.Experimental validation on the MSCOCO dataset shows that the method is able to generate captioning more accurately,and the performance is considerably improved in all evaluation metrics when compared with the most current state-of-the-art image captioning methods.

关 键 词：图像描述 视觉信息 语义信息 时间维度 门控机制 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]