双尺度顺序填充的深度图像修复  

Depth image recovery based on dual-scale sequential optimized filling

作　　者：陈东岳[1] 朱晓明 马腾 宋园园 贾同[1] Chen Dongyue;Zhu Xiaoming;Ma Teng;Song Yuanyuan;Jia Tong(College of Information Secience and Engineering,Northeastern University,Shenyang110819,China)

机构地区：[1]东北大学信息科学与工程学院,沈阳110819

出　　处：《中国图象图形学报》2022年第8期2344-2355,共12页Journal of Image and Graphics

基　　金：国家重点研发计划资助(2017YFC0821402);国家自然科学基金项目(U1613214)。

摘　　要：目的 深度图像作为一种重要的视觉感知数据,其质量对于3维视觉系统至关重要。由于传统方法获取的深度图像大多有使用场景的限制,容易受到噪声和环境影响,导致深度图像缺失部分深度信息,使得修复深度图像仍然是一个值得研究并有待解决的问题。对此,本文提出一种用于深度图像修复的双尺度顺序填充框架。方法 首先,提出基于条件熵快速逼近的填充优先级估计算法。其次,采用最大似然估计实现缺失深度值的最优预测。最后,在像素和超像素两个尺度上对修复结果进行整合,准确实现了深度图像孔洞填充。结果 本文方法在主流数据集MB(Middlebury)上与7种方法进行比较,平均峰值信噪比(peak signal-to-noise ratio, PSNR)和平均结构相似性指数(structural similarity index, SSIM)分别为47.955 dB和0.998 2;在手工填充的数据集MB+中,本文方法的PSNR平均值为34.697 dB,SSIM平均值为0.978 5,对比其他算法,本文深度修复效果有较大优势。在时间效率对比实验中,本文方法也表现优异,具有较高的效率。在消融实验部分,对本文提出的填充优先级估计、深度值预测和双尺度改进分别进行评估,验证了本文创新点的有效性。结论 实验结果表明,本文方法在鲁棒性、精确度和效率方面相较于现有方法具有比较明显的优势。Objective The acquired depth information has led to the research development of three-dimensional reconstruction and stereo vision. However, the acquired depth images issues have challenged of image holes and image noise due to the lack of depth information. The quality of the depth image is as a benched data source for each 3 D-vision(3 DV) system. Our method is focused on the lack of depth map information repair derived from objective factors in the depth acquisition process. It is required of the high precision, the spatial distribution difference between color and depth features, the interference of noise and blur, and the large scale holes information loss. Method Real-time ability is relatively crucial in terms of the depth image recovery algorithms serving as pre-processing modules in the 3 DV systems. The sequential filling method has been optimized in computational speed by processing each invalid point in one loop. The invalid points based pixels are obtained without depth values. By contrast, depth values captured pixels are referred to as valid points. Therefore, we facilitate a dual-scale sequential filling framework for depth image recovery. We carry out filling priority estimation and depth value prediction of the invalid points in this framework. For the evaluation of the priority of invalid points, we use conditional entropy as the benchmark for evaluating the priority of invalid point filling evaluation and verification. It is incredible to estimate the filling priority and filling depth value through the overall features of a single pixel and its 8-neighborhood. However, the use of multi-scale filtering increases the computational costs severely. We introduce the super-pixel over-segmentation algorithm to segment the input image into more small patches, which ensures the pixels inside the super-pixel homogeneous contexts like color, texture, and depth. We believe that the super-pixels can provide more reliable features in larger scale for priority estimation filling and depth value prediction.

关 键 词：深度图像修复 顺序填充 条件熵快速逼近 深度最优预测 超像素 

分 类 号：TN911.73[电子电信—通信与信息系统]