基于区域特征的Transformer混合结构图像藏文描述模型  

Region Features Based Hybrid Structure Transformer for Image Tibetan Captioning

作　　者：周子琦 黄鹤鸣[1,2] 才让当知 夏吾吉 达飞鹏[3] ZHOU Ziqi;HUANG Heming;CaiRang DANGZHI;XIA Wuji;DA Feipeng(School of Computer Science and Technology,Qinghai Normal University,Xining,Qinghai 810008,China;The State Key Laboratory of Tibetan Intelligent Information Processing and Application,Qinghai Normal University,Xining,Qinghai 810008,China;School of Automation,Southeast University,Nanjing,Jiangsu 210096,China)

机构地区：[1]青海师范大学计算机学院,青海西宁810008 [2]青海师范大学藏语智能信息处理及应用国家重点实验室,青海西宁810008 [3]东南大学自动化学院,江苏南京210096

出　　处：《中文信息学报》2025年第1期37-46,55,共11页Journal of Chinese Information Processing

基　　金：国家自然科学基金(62066039);青海省自然科学基金(2022-ZJ-925)。

摘　　要：端到端的Transformer模型在中、英文图像描述数据集上取得了良好的效果,但其巨大的训练数据需求使其在规模较小的藏文数据集上效果并不理想。针对此问题,该文基于编码器—解码器架构的基线模型,引入目标检测网络和自注意力机制以替换CNN和LSTM,提出基于区域特征的Transformer混合结构模型ReT。首先,该模型将Faster-RCNN提取的图像区域特征作为Transformer编码器的输入,将经过向量化后的文本序列特征作为Transformer解码器的输入;其次,Transformer编码器和解码器各自利用自注意力机制对输入向量进行加权;最后,Transformer解码器中的交叉自注意力机制根据文本特征对图像特征进行加权,并以此生成描述。实验表明,在Flickr8k-Ti数据集上以藏文音节为基本单元进行模型的训练和验证时,相比基线模型和端到端的Transformer模型,模型ReT在CIDEr指标上分别提升10.4%和5.3%,取得了更好的效果。The end-to-end Transformer model has achieved good results in Chinese and English image captioning datasets.To adapt to smaller Tibetan datasets,this paper proposes a hybrid structure Transformer model(named ReT)based on region features,which is a two-stage pipeline model with object detection networks and self attention mechanisms.The model takes the image region features extracted by Faster-RCNN as input to the Transformer encoder,and the vectorized text sequence features as input to the Transformer decoder.On the basis of the self-attention mechanism,the cross self-attention mechanism in the Transformer decoder weights image features based on text features and generates descriptions accordingly.The experiments show that,with the Tibetan syllable as the basic unit on the Flickr8k-Ti dataset,the ReT model outperforms the baseline and the end-to-end transformer by 10.4%and 5.3%according to the CIDEr index,respectively.

关 键 词：图像藏文描述 藏文音节 Faster-RCNN TRANSFORMER 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]