基于目标检测和知识增强的图像描述生成  

Image Caption Generation Based on Object Detection and Knowledge Enhancement

作　　者：钟家国 王东升 路曼 李佳伟 ZHONG Jiaguo;WANG Dongsheng;LU Man;LI Jiawei(School of computing,Jiangsu University of Science and Technology,Zhenjiang,Jiangsu 212100,China)

机构地区：[1]江苏科技大学计算机学院,江苏镇江212100

出　　处：《中文信息学报》2024年第11期160-170,共11页Journal of Chinese Information Processing

基　　金：国家自然科学基金(61702234);船舶总体性能创新研究开放基金(25422217)。

摘　　要：现有的图像描述生成方法和结果主要侧重于识别图像中已有的图像内容及其关系,而无法生成具有细粒度背景知识的描述。为克服传统图像描述模型时不能描述图片中深层次语义的问题,一方面,该文提出了一种基于目标检测和知识增强的图像描述生成方法,首先在目标检测阶段,提出了一种融合多维度信息的目标分类检测器(Fusion Target Classification Detector,FTCD),通过该检测器获得图中的人脸、商品及物体等目标的标签;其次,引入知识图谱,利用目标分类检测器得到的目标标签在知识图谱中查询相关知识;最后,目标的标签集合和相关知识共同送入模型进行编码;在模型的解码端引入注意力机制,用于引导模型选取合适的信息后生成图像描述。另一方面,针对MSCOCO数据集中的人工描述缺乏常识知识的问题,该文提出一种引入知识图谱的评测指标SPICE-K,SPICE-K可以评价具有常识知识的图像描述。实验结果表明,与基准LBPF模型相比,该文提出的方法有1%的准确率提升。实验分析表明,与基准模型相比,该文方法的性能提升主要来自于引入知识图谱和该文提出的目标分类检测器。Existing image description generation methods mainly focus on recognizing existing image contents and their relationships in images,without generating descriptions with fine-grained background knowledge.To describe deep semantics in images,this paper proposes an image description generation method based on target detection and knowledge enhancement.Firstly,in the target detection stage,the paper proposes a fusion target classification detector(FTCD)that fuses multidimensional information,through which the labels of targets such as faces,goods and objects in the picture are obtained.Secondly,the knowledge graph is introduced,and the target labels obtained by the target classification detector are used to query the related knowledge in the knowledge graph.Finally,after inputting the set of labels of targets and related knowledge for encoding,an attention mechanism is introduced to guide the model to generate image descriptions after selecting the appropriate information.Meanwhile,we propose an evaluation metric SPICE-K that introduces knowledge graphs to evaluate image descriptions with common-sense knowledge.Experimental results show that the proposed method achieves 1%improvement in accuracy compared with the benchmark LBPF model.

关 键 词：图像描述 目标检测 知识图谱 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]