基于多角度融合与联合记忆网络的视频问答认知模型  

A cognitive model of video QA based on multi-angle fusion and joint memory network

作　　者：倪琴 刘双 余杨泽 林欣[3] 邓赐平[4] NI Qin;LIU Shuang;YU Yangze;LIN Xin;DENG Ciping(School of Education,Shanghai International Studies University,Shanghai 201620,China;College of Information,Mechanical and Electrical Engineering,Shanghai Normal University,Shanghai 201418,China;School of Computer Science and Technology,East China Normal University,Shanghai 200062,China;The School of Psychology and Cognitive Science,East China Normal University,Shanghai 200062,China)

机构地区：[1]上海外国语大学国际教育学院,上海201620 [2]上海师范大学信息与机电工程学院,上海201418 [3]华东师范大学计算机科学与技术学院,上海200062 [4]华东师范大学心理与认知科学学院,上海200062

出　　处：《上海师范大学学报（自然科学版中英文）》2024年第5期596-603,共8页Journal of Shanghai Normal University(Natural Sciences)

基　　金：国家自然科学基金(6210020445);上海市自然科学基金(21ZR1446900,21511100102)。

摘　　要：为了解决现有视频问答模型认知推理能力不足的问题,引入旁观者记忆模块,提出了基于多角度融合与联合记忆网络的机器认知模型.该模型根据问题定位目标对象,获得视频中对应的区域特征,同时联合视频的运动特征和外观特征,通过加入时间注意力机制的门控循环单元,有效地融合问题特征和视频特征,用于答案的生成,以提高模型认知推理能力.实验结果表明:相比于现有的视频问答模型,该模型的准确率更高,尤其对于推理难度较大的信念推理问题,该模型体现出了更好的推理能力及泛化性能.In order to solve the problem of insufficient cognition and reasoning ability in existing video question answering models,an observer memory module was introduced,and a machine cognition model based on multi-angle fusion and joint memory network was proposed.The target object was located based on the problem and the corresponding regional features in the video were obtained by this model.At the same time,the motion and appearance features of the video were combined.By adding a gated loop unit with time attention mechanism,the problem features and video features were integrated more effectively for answer generation,which improved the model’s cognitive reasoning ability.The experimental results showed that compared to existing video QA models,this model had higher accuracy,which demonstrated better reasoning ability and generalization ability especially for belief reasoning problems with greater difficulty in cognitive reasoning task.

关 键 词：认知推理 注意力机制 记忆网络 视频问答 

分 类 号：TP301[自动化与计算机技术—计算机系统结构]