结合时序网络和金字塔融合的稳像修复方法  

Image stabilization repair method combining time series network and pyramid fusion

作　　者：刘清[1,2,3] 李世超[1,2,3] 王文杉 师文喜 成科扬[1,2,3] LIU Qing;LI Shi-chao;WANG Wen-shan;SHI Wen-xi;CHENG Ke-yang(School of Computer Science and Communication Engineering,Jiangsu University,Zhenjiang Jiangsu 212013,China;Cyber Space Security Academy of Jiangsu University,Zhenjiang Jiangsu 212013,China;Jiangsu Province Big Data Ubiquitous Perception and Intelligent Agricultural Application Engineering Research Center,Zhenjiang Jiangsu 212013,China;National Engineering Laboratory for Public Security Risk Perception and Control by Big Data,China Academy of Electronic Sciences,Beijing 100041,China;Xinjiang Lianhaichuangzhi Information Technology Co.,Ltd.,Urumqi Xinjiang 830001,China)

机构地区：[1]江苏大学计算机科学与通信工程学院,江苏镇江212013 [2]江苏大学网络空间安全研究院,江苏镇江212013 [3]江苏省大数据泛在感知与智能农业应用工程研究中心,江苏镇江212013 [4]中国电子科学研究院社会安全风险感知与防控大数据应用国家工程实验室,北京100041 [5]新疆联海创智信息科技有限公司,新疆乌鲁木齐830001

出　　处：《图学学报》2021年第1期65-70,共6页Journal of Graphics

基　　金：国家自然科学基金项目(61972183,61672268);社会安全风险感知与防控大数据应用国家工程实验室主任基金项目(201807)。

摘　　要：针对视频稳像领域内视频图像缺损填充效果不佳,严重影响视觉效果,且导致稳像处理后的视频不稳的黑边填充问题,提出了一种基于时序网络预测和金字塔融合的图像修复方法。首先结合预裁剪机制自适应判断当前帧是否需修复;然后将截止至当前时刻的所有帧送入卷积神经网络(CNN)和门控循环单元(GRU)的模型进行待填充部分的预测;随后采用改进的加权最佳缝合线进行拼接并在高斯拉普拉斯金字塔中进行图像融合重构;最终在重构完成后裁剪尺寸。实验结果表明,该方法平均峰值信噪比(PSNR)相较于对比算法提高了2~5dB,平均结构相似度(SSIM)较对比算法提升了约2%~7%。该方法修复后的视频缺损填充自然,视觉效果较为稳定,即使在黑边面积较大时也有良好的修复效果,可用于多种摄像平台及不同场景下。To address the problems of the poor filling effect of the video image defect in video image stabilization,which seriously affects the visual effect and causes the black edge filling of the video after image stabilization processing,an image repair method was proposed based on time series network prediction and pyramid fusion.First,the pre-cutting mechanism was employed to adaptively determine whether the current frame needed to be repaired.Then all frames up to the current moment were sent to the model combining convolutional neural networks(CNN)and gated recurrent unit(GRU)to predict the part to be filled.Next,the improved weighted optimal stitching was used for stitching and image fusion reconstruction in the Gaussian Laplace pyramid.Finally,the size was cut after the completion of reconstruction.The experimental results show that the average peak signal to noise ratio(PSNR)of the method was 2–5 dB higher than that of the compared algorithm,and that the average structural similarity(SSIM)was improved by about 2%–7%.In addition,the video defect repaired by this method exhibits a natural filling effect and a relatively stable visual effect.Even in the cases of large black areas,the repair performance remains stable,which can be applied to a variety of camera platforms and different scenarios.

关 键 词：视频稳像 视频图像修复 时序网络 金字塔融合 最佳缝合线 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]