面向虚拟视点绘制空洞填充的渐进式迭代网络  

Progressive iteration network for hole filling in virtual view rendering

作　　者：刘家希 周洋[1] 林坤 殷海兵 唐向宏[1] Liu Jiaxi;Zhou Yang;Lin Kun;Yin Haibing;Tang Xianghong(School of Communication Engineering,Hangzhou Dianzi University,Hangzhou 310018,China)

机构地区：[1]杭州电子科技大学通信工程学院,杭州310018

出　　处：《中国图象图形学报》2024年第7期1948-1959,共12页Journal of Image and Graphics

基　　金：浙江省自然科学基金项目(LY21F020021);浙江省尖兵领雁计划项目(2022C01068)。

摘　　要：目的 基于深度图像的绘制(depth image based rendering,DIBR)是合成虚拟视点图像的关键技术,但在绘制过程中虚拟视图会出现裂纹和空洞问题。针对传统算法导致大面积空洞区域像素混叠和模糊的问题,将深度学习模型应用于虚拟视点绘制空洞填充领域,提出了面向虚拟视点绘制空洞填充的渐进式迭代网络。方法 首先,使用部分卷积对大面积空洞进行渐进修复。然后采用U-Net网络作为主干对空洞区域进行编解码操作,同时嵌入知识一致注意力模块加强网络对有效特征的利用。接着通过加权合并方法来融合每次渐进式迭代生成的特征图,保护早期特征不被破坏。最后结合上下文特征传播损失提高网络匹配过程中的鲁棒性。结果 在微软实验室提供的2个多视点3D(three-dimension)视频序列以及4个3D-HEVC(3D high efficiency video coding)序列上进行定量与定性评估实验,以峰值信噪比(peak signal-to-noise ratio,PSNR)和结构相似性(structural similarity,SSIM)作为指标。实验结果表明,本文算法在主观和客观上均优于已有方法。相比于性能第2的模型,在Ballet、Breakdancers、Lovebird1和Poznan_Street数据集上,本文算法的PSNR提升了1.302 dB、1.728 dB、0.068 dB和0.766 dB,SSIM提升了0.007、0.002、0.002和0.033;在Newspaper和Kendo数据集中,PSNR提升了0.418 dB和0.793 dB,SSIM提升了0.011和0.007。同时进行消融实验验证了本文方法的有效性。结论 本文提出的渐进式迭代网络模型,解决了虚拟视点绘制空洞填充领域中传统算法过程烦琐和前景纹理渗透严重的问题,取得了极具竞争力的填充结果。Objective Depth image-based rendering(DIBR) makes full use of the depth information in a reference image and can combine color image and depth information organically,which is faster and less complex than the general rendering method.Therefore,DIBR is selected by ISO as the primary virtual view rendering technology in 3D multimedia video.The principal challenge associated with virtual view rendering technology is the 3D warping of the reference view,which leads to exposure of the background that was previously obstructed by the foreground.As a result,certain areas appear as holes in the virtual view due to the absence of pixel values.The search for an effective solution to address missing regions in the rendered view image is a critical challenge in virtual view rendering technology.The traditional algorithms mainly fill the holes based on the space-domain consistency and time-domain consistency methods.Filtering can effectively remove the cracks and some of the holes but cannot handle the large-area holes.The patch-based method can fill large-area holes,but the process is tedious,the amount of data is too large,and the accuracy of searching for the best matching patch is not high,which may lead to the texture belonging to the foreground being incorrectly filled to the hole area belonging to the background.Based on the time-domain consistency method,a model is developed to reconstruct the vacant part of the background using various models,and the foreground part is repositioned to the virtual viewpoint location to reduce the computational complexity and increase the adaptability to the scene.However,the moving camera scene contains both stationary and moving objects,which easily causes some parts of the foreground to be modeled as the background,resulting in the mixing of foreground and background pixels.Therefore,a deep learning model is applied to the field of hole filling in virtual view rendering,and a progressive iterative network for hole filling in virtual view rendering is proposed to address the problem

关 键 词：虚拟视点绘制 空洞填充 注意力 特征提取 多视点视频加深度 

分 类 号：TN919.8[电子电信—通信与信息系统]