一种多模态跨媒体检索的融媒体影视系统  

A film and television media convergence system based on multi modal cross-media retrieval

作　　者：李春芳 刘永久 王楷翔 杨睿 张凌飞 李敏 邓智铭 石民勇[1] LI Chunfang;LIU Yongjiu;WANG Kaixiang;YANG Rui;ZHANG Lingfei;LI Min;DENG Zhiming;SHI Minyong(School of Computer and Cyber Sciences,Communication University of China,Beijing 100024,China)

机构地区：[1]中国传媒大学计算机与网络空间安全学院,北京100024

出　　处：《中国传媒大学学报（自然科学版）》2021年第4期63-71,共9页Journal of Communication University of China：Science and Technology

基　　金：国家社科基金艺术学项目资助(18BC034);中央高校基本科研业务费资助(CUC210A008)。

摘　　要：视频是最有影响力的传播媒介,然而其非线性检索仍然困难。本文创新性工作包括:基于图像识别提取字幕,基于卷积神经网络识别人脸,通过字幕和人脸解决了影视视频的非线性检索问题;从字幕文本提取重要实体,用海量知识库和电子书补全影视关联知识,构建了文本、电子书和视频融合的跨媒体应用;以字幕词云和人物实体词云,实现影视的概览理解和检索导航;以众包实现字幕、电子书、人脸和实体信息的修正。以近代史献礼电影、中国诗词大会和科技纪录片为例系统完整地实现了一个示范性融媒体影视系统。Video is the most influential media,but it's difficult to nonlinearly search video content.The creative work of this paper includes:Based on image processing to recognize video subtitle and convolutional neural networks to recognize faces of characters,the problem of film and TV video nonlinear retrieval is solved.Further,we extract important entities from subtitle text and enhance their relevant knowledge with large scale knowledge base and e-books,which constructs a cross-media application system of video,text,and e-book.Word cloud of subtitles and character entities are designed to facilitate video overview understanding and navigating retrieval.Crowdsourcing technology is used to update the amendments of subtitles,e-books,face recognition and entities information.A typical cross media convergence system are completely implemented including movies in modern history,conference of the Chinese poetry,and information technology documentary video.

关 键 词：融媒体 跨媒体检索 字幕识别 人脸识别 

分 类 号：TP302[自动化与计算机技术—计算机系统结构]