基于深度学习的行为识别多模态融合方法综述  

Survey on Multi-modality Fusion Methods for Action Recognition Based on Deep Learning

作　　者：詹健浩 吴鸿伟[2] 周成祖[2] 陈晓筹 李晓潮 ZHAN Jian-Hao;WU Hong-Wei;ZHOU Cheng-Zu;CHEN Xiao-Chou;LI Xiao-Chao(Department of Microelectronics and Integrated Circuit,Xiamen University,Xiamen 361005,China;Xiamen Meiya Pico Information Co.Ltd.,Xiamen 361016,China;Information and Network Center,Xiamen University,Xiamen 361005,China)

机构地区：[1]厦门大学电子科学与技术学院,厦门361005 [2]厦门市美亚柏科信息股份有限公司,厦门361016 [3]厦门大学信息与网络中心,厦门361005

出　　处：《计算机系统应用》2023年第1期41-49,共9页Computer Systems & Applications

基　　金：福建省高校产学研联合创新项目(2022H6004);集成电路设计与测试分析福建省高校重点实验室基金;厦门大学马来西亚研究基金(XMUMRF/2019-C4/IECE/0008)。

摘　　要：行为识别是通过对视频数据进行处理分析从而让计算机理解人的动作和行为.不同模态数据在外观、姿态、几何、光照和视角等主要特征上各有优势,通过多模态融合将这些特征进行融合可以获得比单一模态数据更好的识别效果.本文对现有行为识别多模态融合方法进行介绍,对比了它们之间的特点以及获得的性能提升,包括预测分数融合、注意力机制、知识蒸馏等晚期融合方法,以及特征图融合、卷积、融合结构搜索、注意力机制等早期融合方法.通过这些分析和比较归纳出未来多模态融合的研究方向.Action recognition aims to make computers understand human actions by the processing and analysis of video data.As different modality data have different strengths in the main features such as appearance,gesture,geometric shapes,illumination,and viewpoints,action recognition based on the multi-modality fusion of these features can achieve better performance than the recognition based on single modality data.In this study,a comprehensive survey of multimodality fusion methods for action recognition is given,and their characteristics and performance improvements are compared.These methods are divided into the late fusion methods and the early fusion methods,where the former includes prediction score fusion,attention mechanisms,and knowledge distillation,and the latter includes feature map fusion,convolution,fusion architecture search,and attention mechanisms.Upon the above analysis and comparison,the future research directions are discussed.

关 键 词：行为识别 深度学习 多模态融合 晚期融合 早期融合 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术] TP18[自动化与计算机技术—计算机科学与技术]