伪时空图卷积网络修复姿态引导的Transformer行人视频修复方法  

Transformer-Based Pedestrian Video Inpainting Guided by Pseudo-Spatiotemporal Pose Correction Graph Convolutional Networks

作　　者：唐福梅 聂勇伟 余嘉祺 张青[2] 李桂清[1] Tang Fumei;Nie Yongwei;Yu Jiaqi;Zhang Qing;Li Guiqing(South China University of Technology,School of Computer Science and Engineering,Guangzhou 510006;Sun Yat-Sen University,School of Computer Science and Engineering,Guangzhou 510006)

机构地区：[1]华南理工大学计算机科学与工程学院,广州510006 [2]中山大学计算机学院,广州510006

出　　处：《计算机辅助设计与图形学学报》2024年第4期552-564,共13页Journal of Computer-Aided Design & Computer Graphics

基　　金：国家自然科学基金面上项目(62072191,61972160);广东省自然科学基金面上项目(2019A1515010860,2021A1515012301)。

摘　　要：为解决监控视频中被遮挡行人的修复问题,提出了一种基于人体姿态的行人视频修复方法,即先修复视频中残缺的行人姿态序列,然后在修补后的姿势序列的引导下修复视频帧中人体的缺失部分.该方法采用OpenPose从视频中提取被遮挡的人体姿态序列,针对其因存在遮挡情况导致未识别出和未准确识别部分关节点的问题,提出了一种伪时空图卷积网络模型对缺失姿态进行修复,得到一个相对准确的姿态序列;基于修复后的姿态,提出了基于姿态序列引导的Transformer行人视频修复模型.在Human3.6M数据集上进行了测试,所提出的方法在4个指标PSNR,RMSE,SSIM,LPIPS上均比对比方法有提升,特别是RMSE指标提升了9.50%,LPIPS指标提升了21.67%.In order to solve the problem of repairing occluded pedestrians in surveillance videos,a pedestrian video inpainting method based on human pose is proposed,which repairs the incomplete pedestrian pose sequence at first,and then inpaints the video frames under the guidance of the repaired pose sequence.Firstly,the proposed method uses OpenPose to extract the occluded human pose sequence from the video.Due to occlusions,some joints of the extracted poses may be unrecognized or inaccurately recognized.We thus propose a pseudo-spatiotemporal graph convolutional network to repair the extracted poses and obtain an accurate pose sequence.We then propose a Transformer-based pedestrian video repair model guided by the repaired pose sequence.Tested on the Human3.6M dataset,the proposed method is better than previous approaches in terms of four metrics including PSNR,RMSE,SSIM,and LPIPS.Especially,RMSE is improved by 9.50%,and LPIPS is improved by 21.67%.

关 键 词：深度学习 图卷积神经网络 TRANSFORMER 人体姿态补全 视频修复 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]