融合时空图卷积的多人交互行为识别  

Multi-person interaction action recognition based onspatio-temporal graph convolution

作　　者：成科扬[1,2,3] 吴金霞 王文杉 荣兰 詹永照 Cheng Keyang;Wu Jinxia;Wang Wenshan;Rong Lan;Zhan Yongzhao(School of Computer Science and Telecommunications Engineering,Jiangsu University,Zhenjiang 212013,China;Jiangsu Province Big Data Ubiquitous Perception and Intelligent Agricultural Application Engineering Research Center,Zhenjiang 212013,China;Cyber Space Security Academy of Jiangsu University,Zhenjiang 212013,China;National Engineering Laboratory for Public Security Risk Perception and Control by Big Data,China Acadeemy of Electronic Sciences,Beijing 100041,China)

机构地区：[1]江苏大学计算机科学与通信工程学院,镇江212013 [2]江苏省大数据泛在感知与智能农业应用工程研究中心,镇江212013 [3]江苏大学网络空间安全研究院,镇江212013 [4]中国电子科学研究院社会安全风险感知与防控大数据应用国家工程实验室,北京100041

出　　处：《中国图象图形学报》2021年第7期1681-1691,共11页Journal of Image and Graphics

基　　金：国家自然科学基金项目(61972183);社会安全风险感知与防控大数据应用国家工程实验室主任基金项目。

摘　　要：目的多人交互行为的识别在现实生活中有着广泛应用。现有的关于人类活动分析的研究主要集中在对单人简单行为的视频片段进行分类,而对于理解具有多人之间关系的复杂人类活动的问题还没有得到充分的解决。方法针对多人交互动作中两人肢体行为的特点,本文提出基于骨架的时空建模方法,将时空建模特征输入到广义图卷积中进行特征学习,通过谱图卷积的高阶快速切比雪夫多项式进行逼近。同时对骨架之间的交互信息进行设计,通过捕获这种额外的交互信息增加动作识别的准确性。为增强时域信息的提取,创新性地将切片循环神经网络(recurrent neural network,RNN)应用于视频动作识别,以捕获整个动作序列依赖性信息。结果本文在UT-Interaction数据集和SBU数据集上对本文算法进行评估,在UT-Interaction数据集中,与H-LSTCM(hierarchical long short-term concurrent memory)等算法进行了比较,相较于次好算法提高了0.7%,在SBU数据集中,相较于GCNConv(semi-supervised classification with graph convolutional networks)、RotClips+MTCNN(rotating cliips+multi-task convolutional neural netowrk)、SGC(simplifying graph convolutional)等算法分别提升了5.2%、1.03%、1.2%。同时也在SBU数据集中进行了融合实验,分别验证了不同连接与切片RNN的有效性。结论本文提出的融合时空图卷积的交互识别方法,对于交互类动作的识别具有较高的准确率,普遍适用于对象之间产生互动的行为识别。Objective The recognition of multi-person interaction behavior has wide applications in real life. At present, human activity analysis research mainly focuses on classifying video clips of behaviors of individual persons, but the problem of understanding complex human activities with relationships between multiple people has not been resolved. When performing multi-person behavior recognition, the body information is more abundant and the description of the two-person action features are more complex. The problems such as complex recognition methods and low recognition accuracy occur easily. When the recognition object changes from a single person to multiple people, we not only need to pay attention to the action information of each person but also need to notice the interaction information between different subjects. At present, the interaction information of multiple people cannot be extracted well. To solve this problem effectively, we propose a multi-person interaction behavior-recognition algorithm based on skeleton graph convolution.Method The advantage of this method is that it can fully utilize the spatial and temporal dependence information between human joints. We design the interaction information between skeletons to discover the potential relationships between different individuals and different key points. By capturing the additional interaction information, we can improve the accuracy of action recognition. Considering the characteristics of multi-person interaction behavior, this study proposes a spatio-temporal graph convolution model based on skeleton. In terms of space, we have various designs for single-person and multi-person connections. We design the single-person connection within each frame. Apart from the physical connections between the points of the body, some potential correlations are also added between joints that represent non-physical connections such as the left and right hands of a single person. We design the interaction connection between two people within each frame. We use

关 键 词：动作识别 交互信息 时空建模 图卷积 切片循环神经网络(RNN) 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]