基于双向特征金字塔的密集视频描述生成方法  

Dense video captioning with bilateral feature pyramid net

作　　者：张煜杨 刘茂福[1,2] ZHANG Yuyang;LIU Maofu(School of Computer Science and Technology,Wuhan University of Science and Technology,Wuhan 430065,China;Hubei Provincial Key Laboratory of Intelligent Information Processing and Real-time Industrial System(Wuhan University of Science and Technology),Wuhan 430081,China)

机构地区：[1]武汉科技大学计算机科学与技术学院,武汉430065 [2]智能信息处理与实时工业系统湖北省重点实验室(武汉科技大学),武汉430081

出　　处：《中国科技论文》2024年第2期200-208,共9页China Sciencepaper

摘　　要：针对当前方法普遍存在较为严重的细节结构信息丢失与事件间重叠的问题,提出一种基于双向特征金字塔的密集视频描述生成方法(dense video captioning with bilateral feature pyramid net,BFPVC)。BFPVC通过带有自底向上、自顶向下、横向链接3条分支的双向特征金字塔强化视频多尺度特征图,兼顾对时序信息、空间信息、语义信息的特征表示,解码器从强化后的视频特征中捕获更加全面的事件候选集,从而为对应的视频事件生成更加丰富、详尽的文本描述。在ActivityNet Captions数据集和YouCook2数据集上的实验结果表明,BFPVC与同类模型相比生成的文本描述更详细、丰富,验证了双向特征金字塔在密集视频描述领域的有效性。In order to solve the generally existing problem of detail structure information loss and inter-event overlap in current dense video description generation method,an end-to-end dense video captioning with bilateral feature pyramid net(BFPVC)was pro-posed.BFPVC strengthened the multi-scale feature maps of videos by incorporating a bidirectional feature pyramid with three branches of bottom-up,top-down,and lateral connections.It balanced the feature representation of temporal,spatial,and semantic information.The backbone network captured a more comprehensive set of event candidates from the enhanced video features,thereby generating richer and more detailed captions for the corresponding video events.The results of this experiments on the ActivityNet Captions dataset and YouCook2 dataset demonstrate that the BFPVC method generates more detailed and richer cap-tions than similar frameworks,validating the effectiveness of the bilateral feature pyramid net in the field of dense video captioning.

关 键 词：密集视频描述 视频描述 视频理解 特征金字塔 自然语言处理 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]