多模态时空特征表示及其在行为识别中的应用  

Multimodal spatial-temporal feature representation and its application in action recognition

作　　者：施海勇 侯振杰[1] 巢新 钟卓锟 Shi Haiyong;Hou Zhenjie;Chao Xin;Zhong Zhuokun(School of Computer and Artificial Intelligence,Changzhou University,Changzhou 213164,China)

机构地区：[1]常州大学计算机与人工智能学院,常州213164

出　　处：《中国图象图形学报》2023年第4期1041-1055,共15页Journal of Image and Graphics

基　　金：国家自然科学基金项目(61063021);江苏省研究生科研创新计划项目(KYCX21_2835)。

摘　　要：目的在人体行为识别研究中,利用多模态方法将深度数据与骨骼数据相融合,可有效提高动作的识别率。针对深度图像信息数据量大、冗余度高等问题,提出一种通过获取关键时程信息动作帧序列降低冗余的算法,即质心运动路径松弛算法,并根据不同模态数据的特点,提出一种新的时空特征表示方法。方法质心运动路径松弛算法根据质心在相邻帧之间的运动距离,计算图像差分后获得的活跃部分的相似系数,然后剔除掉相似度高的帧,获得足以表达行为的关键时程信息。根据图像动态部分的变化特性、人体各部分在运动中的协同性和局部显著性特征构建一种新的时空特征表示方法。结果在MSR-Action3D数据集上对本文方法的效果进行验证。在3个子集中进行交叉验证的平均分类识别率为95.7432%,分别比Multi-fused,CovP3DJ,D3D-LSTM(densely connected 3DCNN and long short-term memory),Joint Subset Selection方法高2.4432%,4.7632%,0.3432%,0.2132%。本文方法在使用完整数据集的扩展实验中进行交叉验证的分类识别率为93.0403%,具有很好的鲁棒性。结论实验结果表明,本文提出的去冗余算法在降低冗余后提升了识别效果,提取的特征之间具有相关性低的特点,在组合识别中具有良好的互补性,有效提高了分类识别的精确度。Objective Human body motion-related recognition has been developing in the context of computer vision and pattern recognition like auxiliary human-computer interaction,motion analysis,intelligent monitoring,and virtual reality.To obtain two-dimensional information for its behavioral recognition,conventional motion behavior recognition is mainly used the RGB image sequence captured by RGB camera.To improve the ability to detect short-duration fragments,current feature descriptors for RGB image sequences are employed to characterize human behavior,such as histogram of oriented gradient(HOG),histogram of optical flow(HOF),and a three-dimensional feature pyramid.Some researchers are focused on the feature that image depth is insensitive to ambient light since RGB images are oriented to behavior image sequences of objects in terms of two-dimensional information.The depth information of the image is coordinated with the features of RGB image to describe the related behavior.Human behavior recognition-relevant multi-modal method can be used to fuse depth data and skeleton data,which can improve the recognition rate of action effectively.Recent depth map is widely used in relevant to human behavior recognition.But,the collection of depth information data is required to be opti⁃mized because of time complexity of feature extraction and space complexity of feature storage.To resolve the problems,we develop an algorithm to optimize frames of the depth map and resource consumption.At the same time,a new representa⁃tion of motion features is facilitated as well according to the motion information of the centroid.Method First,the temporal feature vector is used in terms of depth map sequence-extracted time sequence information.The centroid motion path relax⁃ation algorithm is used to realize depth image de-duplication and de redundancy,and the skeleton map-extracted spatial structure feature vector from are spliced to form the spatio-temporal feature input.Next,spatial features are extracted in terms of the original skel

关 键 词：行为识别 质心运动 关键时程信息 时空特征表示 多模态融合 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]