基于CLIP的多模态融合视频描述生成  

Multimodal fusion video caption generation based on CLIP

作　　者：王亮 夏舟勇[1,2] 胡营营 王军 WANG Liang;XIA Zhou-yong;HU Ying-ying;WANG Jun(College of Computer Science and Technology,Shenyang University of Chemical Technology,Shenyang 110142,China;Liaoning Provincial Key Laboratory of Intelligent Technology for Chemical Process Industry,Shenyang University of Chemical Technology,Shenyang 110142,China)

机构地区：[1]沈阳化工大学计算机科学与技术学院,辽宁沈阳110142 [2]沈阳化工大学辽宁省化工过程工业智能化技术重点实验室,辽宁沈阳110142

出　　处：《计算机工程与设计》2025年第2期384-391,共8页Computer Engineering and Design

基　　金：国家外国专家项目计划基金项目(G2022006008L);中国高校产学研创新基金项目(2021LD06009);辽宁省自然科学基金项目(2022-MS-291);辽宁省教育厅科研基金项目(LJ2020024);辽宁省教育厅基本科研基金项目(LJKMZ20220781);辽宁省教育厅基本科研面上基金项目(JYTMS20231488)。

摘　　要：为解决视频描述任务中2D的CLIP预训练模型缺乏时序关系与动作检测敏感性以及信息冗余问题,提出一种基于CLIP的结合注意力掩码与运动表示增强的多模态融合视频描述模型。采用可学习令牌整理冻结的CLIP特征、运动特征与音频特征中的关键信息,优化多模态融合;引入关键词检测任务,提高关键信息提取能力;采用基于相关度的多头注意力掩码机制解决冗余问题;利用CLIP特征的向量差变换增强运动表示。实验结果表明,该模型性能优于现有视频描述生成方法,CIDEr指标在MSR-VTT数据集上提升了2.33%,在VATEX数据集上提升了3.12%。To address the problems of lack of temporal relationship and motion detection sensitivity,as well as information redundancy in 2D CLIP pre-training models for video caption task,a multimodal fusion video caption model based on CLIP combined with attention mask and motion representation enhancement was proposed.Learnable tokens were used to sort out the key information in frozen CLIP features,motion features and audio features,optimizing multimodal fusion.The keyword detection task was introduced to improve the ability to extract key information.The multi-head attention mask mechanism based on correlation was used to solve the redundancy problem.The vector difference transformation of CLIP features was used to enhance motion representation.Experimental results show that the model performs better than existing video caption generation methods,CIDEr index is improved by 2.33%on MSR-VTT and 3.12%on VATEX.

关 键 词：预训练模型 视频描述 多模态 特征融合 运动表示 注意力掩码 关键词检测 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]