用于视频修复的连贯语义时空注意力网络  

Coherent Semantic Spatial-Temporal Attention Network for Video Inpainting

作　　者：刘浪 李梁[1] 但远宏[1] LIU Lang;LI Liang;DAN Yuan-hong(College of Computer Science and Engineering,Chongqing University of Technology,Chongqing 400054,China)

机构地区：[1]重庆理工大学计算机科学与工程学院,重庆400054

出　　处：《计算机科学》2021年第10期239-245,共7页Computer Science

基　　金：国防科技创新特区项目。

摘　　要：现有的视频修复方法通常会产生纹理模糊、结构扭曲的内容以及伪影,而将基于图像的修复模型直接应用于视频修复会导致时间上的不一致。从时间角度出发,提出了一种新的用于视频修复的连贯语义时空注意力(Coherent Semantic Spatial-Temporal Attention,CSSTA)网络,通过注意力层,使得模型关注于目标帧被遮挡而相邻帧可见的信息,以获取可见的内容来填充目标帧的孔区域(hole region)。CSSTA层不仅可以对孔特征之间的语义相关性进行建模,还能对远距离信息和孔区域之间的远程关联进行建模。为合成语义连贯的孔区域,提出了一种新的损失函数特征损失(Feature Loss)以取代VGG Loss。模型建立在一个双阶段粗到精的编码器-解码器结构上,用于从相邻帧中收集和提炼信息。在YouTube-VOS和DAVIS数据集上的实验结果表明,所提方法几乎实时运行,并且在修复结果、峰值信噪比(PSNR)和结构相似度(SSIM)3个方面均优于3种代表性视频修复方法。Existing video inpainting methods usually produce blurred texture,distorted structure and artifacts,while applying image-based inpainting model directly to the video inpainting will lead to inconsistent time.From the perspective of time,a novel coherent semantic spatial-temporal attention(CSSTA)for video inpainting is proposed,through the attention layer,the model focuses on the information that the target frame is partially blocked and the adjacent frames are visible,so as to obtain the visible content to fill the hole region of the target frame.The CSSTA layer can not only model the semantic correlation between hole features but also remotely correlate the long-range information with the hole regions.In order to complete semantically coherent hole regions,a novel loss function Feature Loss is proposed to replace VGG Loss.The model is built on a two-stage coarse-to-fine encoder-decoder model for collecting and refining information from adjacent frames.Experimental results on the YouTube-VOS and DAVIS datasets show that the method in this paper runs almost in real-time and outperforms the three typical video inpainting methods in terms of inpainting results,peak signal-to-noise ratio(PSNR)and structural similarity(SSIM).

关 键 词：视频修复 图像修复 时空注意力 特征损失 VGG Loss 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]