基于CLIP生成多事件表示的视频文本检索方法  

CLIP Based Multi-Event Representation Generation for Video-Text Retrieval

作　　者：涂荣成 毛先领[1] 孔伟杰 蔡成飞 赵文哲 王红法 黄河燕[1] Tu Rongcheng;Mao Xianling;Kong Weijie;Cai Chengfei;Zhao Wenzhe;Wang Hongfa;and Huang Heyan(Department of Computer Science and Technology,Beijing Institute of Technology,Beijing 100081;School of Information Engineering,Peking University,Shenzhen,Guangdong 518055;School of Electronic Information,Zhejiang University,Hangzhou 310058;School of Software,South China University of Technology,Guangzhou 510006;Institute of Automation,Chinese Academy of Sciences,Beijing 100190)

机构地区：[1]北京理工大学计算机学院,北京100081 [2]北京大学信息工程学院,广东深圳518055 [3]浙江大学电子信息学院,杭州310058 [4]华南理工大学软件学院,广州510006 [5]中国科学院自动化所,北京100190

出　　处：《计算机研究与发展》2023年第9期2169-2179,共11页Journal of Computer Research and Development

基　　金：国家重点研发计划项目(2018YFB1005100);国家自然科学基金项目(U21B2009,62172039,61732005,61602197,L1924068);北京语言资源先进创新中心基金项目(TYZ19005)。

摘　　要：视频-文本检索作为一项被广泛应用于现实生活中的多模态检索技术受到越来越多的研究者的关注.近来,大部分视频文本工作通过利用大规模预训练模型中所学到的视觉与语言之间的匹配关系来提升文本视频间跨模态检索效果.然而,这些方法忽略了视频、文本数据都是由一个个事件组合而成.倘若能捕捉视频事件与文本事件之间的细粒度相似性关系,将能帮助模型计算出更准确的文本与视频之间的语义相似性关系,进而提升文本视频间跨模态检索效果.因此,提出了一种基于CLIP生成多事件表示的视频文本检索方法(CLIP based multi-event representation generation for video-text retrieval,CLIPMERG).首先,通过利用大规模图文预训练模型CLIP的视频编码器(ViT)以及文本编码器(Tansformer)分别将视频、文本数据转换成视频帧token序列以及文本的单词token序列;然后,通过视频事件生成器(文本事件生成器)将视频帧token序列(单词token序列)转换成k个视频事件表示(k个文本事件表示);最后,通过挖掘视频事件表示与文本事件表示之间的细粒度关系以定义视频、文本间的语义相似性关系.在3个常用的公开视频文本检索数据集MSR-VTT,DiDeMo,LSMDC上的实验结果表明所提的CLIPMERG优于现有的视频文本检索方法.Video-text retrieval has been widely used in many real-world applications and attracted more and more research attention.Recently,many work has been proposed to leverage the visual-language matching knowledge of the pre-training models to further improve the retrieval performance.However,these methods ignore that video and text data are composed of events.If the fine-grained similarities between events in video and events in text can be captured well,it will help to calculate more accurate semantic similarities between texts and videos,and then improve the retrieval performance.Hence,in this paper,we propose a CLIP based multi-event representation generation for video-text retrieval,called CLIPMERG.Specifically,CLIPMERG first utilizes the video encoder and text encoder of pre-training model CLIP to transform the video and text inputs into video frame token sequences and word token sequences,respectively.Next,CLIPMERG uses a video(text)event generator to map the video frame(text word)token sequence into k video(text)event representations.Finally,CLIPMERG calculates the semantic similarities between videos and texts through capturing the fine-grained similarities between video event representations and text event representations.Extensive experimental results on three widely used benchmark datasets MSR-VTT,DiDeMo and LSMDC show that our proposed CLIPMERG achieves better performance than state-of-the-art baselines on the video-text retrieval tasks.

关 键 词：预训练模型 视频文本检索 事件表示 CLIP模型 Transformer模型 

分 类 号：TP183[自动化与计算机技术—控制理论与控制工程]