基于注意力机制的多模态人体行为识别算法  

Multi-modal Human Behavior Recognition Algorithm Based on Attention Mechanism

作　　者：宋真东 杨国超 马玉鹏 冯晓毅[1] SONG Zhendong;YANG Guochao;MA Yupeng;FENG Xiaoyi(School of Electronics and Information,Northwestern Polytechnical University,Xi'an 710129,China;Shanxi Huaming Putai Medical Equipment,Xi'an 710119,China;College of Computer and Cyber Security,Hebei Normal University,Shijiazhuang 050024,China)

机构地区：[1]西北工业大学电子信息学院,西安710129 [2]陕西华明普泰医疗设备有限公司,西安710119 [3]河北师范大学计算机与网络空间安全学院,石家庄050024

出　　处：《计算机测量与控制》2022年第2期276-283,共8页Computer Measurement &Control

基　　金：陕西省重点研发项目(2021ZDLGY15-01,2021ZDLGY09-04,2021GY-004和2020GY-050);深圳市国际合作研究项目(GJHZ20200731095204013);国家自然基金(61772419)。

摘　　要：提出了基于注意力机制的多模态人体行为识别算法;针对多模态特征的有效融合问题,设计基于注意力机制的双流特征融合卷积网络(TAM3DNet,two-stream attention mechanism 3D network);主干网络采用结合注意力机制的注意力3D网络(AM3DNet,attention mechanism 3D network),将特征图与注意力图进行加权后得到加权行为特征,从而使网络聚焦于肢体运动区域的特征,减弱背景和肢体静止区域的影响;将RGB-D数据的颜色和深度两种模态数据分别作为双流网络的输入,从两条分支网络得到彩色和深度行为特征,然后将融合特征进行分类得到人体行为识别结果。A multi-modal human behavior recognition algorithm based on the attention mechanism is proposed.Aiming at effective fusion problem of the multimodal features,a two-stream feature fusion convolutional two-stream attention mechanism 3D network(TAM3Dnet)based on the attention mechanism is designed.The backbone network adopts the attention mechanism 3D network(AM3DNet),which combines with the attention mechanism,and weights the feature and attention map to obtain the weighted behavior characteristics,so that the network focuses on the characteristics of the limb movement area and reduces the influence of the background and limb rest area.The color and depth modal data for the RGB-D are respectively used as the input of the dual-stream network,and the color and depth behavior features are obtained from two branch networks,and then the fusion features are classified to obtain the human behavior recognition results.

关 键 词：RGB-D图像 多模态特征 人体行为 双流网络 注意力机制 特征融合 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]