MASR-PSN:低分光度立体图像的高分法向重建深度学习模型  

MASR-PSN:a low-resolution photometric stereo images-relevant deep learning model for high-resolution surface normal reconstruction

作　　者：举雅琨 蹇木伟 饶源 张述[1] 高峰[1] 董军宇[1] Ju Yakun;Jian Muwei;Rao Yuan;Zhang Shu;Gao Feng;Dong Junyu(School of Computer Science and Technology,Ocean University of China,Qingdao 266100,China;School of Computer Science and Technology,Shandong University of Finance and Economics,Jinan 250014,China)

机构地区：[1]中国海洋大学计算机科学与技术学院,青岛266100 [2]山东财经大学计算机科学与技术学院,济南250014

出　　处：《中国图象图形学报》2023年第7期2120-2134,共15页Journal of Image and Graphics

基　　金：国家自然科学基金项目(61501417,61976123,61601427,41906177);国家重大科研仪器研制项目(41927805)。

摘　　要：目的光度立体算法是一种单视角下的稠密三维重建方法,其利用相同视角下来自不同光照方向的一系列图像恢复像素级的表面法向。拍摄光度立体图像所用的高分辨率线性响应相机的成本十分昂贵且难以获取,很难通过传感器直接获取超高分辨率图像来恢复高分辨率表面法向。因此,提出一种基于深度神经网络的光度立体超分算法,以从低分光度立体图像中恢复出准确的高分表面法向。方法首先,对原始的低分光度立体图像进行归一化预处理操作,以消除剧烈变化的表面反射率影响,并消减过饱和镜面反射的影响。随后,提出多层聚合超分光度立体网络(multi-level aggregation super resolution photometric stereo network,MASR-PSN)。MASR-PSN包含一个新颖的深浅层融合的最大池化聚合框架、权值共享的特征回归器、并行设计的不同尺寸卷积核的并行回归器结构,能够在保留多尺度信息的同时,增强特征表示,防止模式坍塌学习到某一固定尺度相关的非重要特征,以及防止3×3卷积核带来空间域上的过度平滑。结果广泛的消融实验证明了提出的深浅层聚合层和并行权值共享回归器的有效性,能明显减少生成表面法向的平均角度误差(mean angular error,MAE)。本文方法仅需其他方法一半分辨率的光度立体图像,而能准确地恢复出复杂表面的结构。DiLiGenT benchmark数据集的定量实验和Light Stage Data Gallery数据集、Gourd数据集的定性实验显示,MASR-PSN在预测表面法向精确度方面有明显提升。在DiLiGenT benchmark数据集中,本文方法在仅使用其他方法一半分辨率的光度立体图像的情况下,以96幅图像为输入时,取得7.31°的平均角度误差,比最佳方法提升0.08°,以10幅图像为输入时,取得9.00°的平均角度误差,比最佳方法提升0.43°。结论提出的MASR-PSN方法提升了光度立体任务表面法向重建的准确性,在低分辨率的�Objective Three-dimensional(3D)reconstruction is currently focused on in computer vision.To optimize the problem of recovering fine details of the surface and dense reconstruction,a fixed scene-related photometric stereo technique can be used in terms of the pixel-wise surface normal under the circumstance of varying shading cues.It can recover per-pixel dense surface normal and improve weak texture-reconstructed objects to a certain extent beyond binocular and multi-view stereo in triangulate sparse 3D points.Photometric stereo can be used in the commonly-used high-precision 3D reconstruction domains like cultural relic reconstruction and industrial defect detection.To solve the complex threedimensional structure and alleviate the blur problem in the normal reconstruction,high-resolution surface normal can provide richer and more effective 3D information.However,due to the high-resolution linear response cameras are high involved,it is still challenged to recover high-resolution surface normal for photometric stereo images.Therefore,it is urgent to develop the high-resolution surface normal reconstruction in terms of low-resolution photometric stereo images analysis.Method We facilitate deep learning based super-resolution photometric stereo algorithm further to recover accurate high-resolution surface normal from low-resolution photometric stereo images.First,a normalized operation is employed to normalize in situ pixels in completed low-resolution photometric stereo images,which can alleviate the effectscontextual of severely changing surface reflectance and oversaturated specular reflection.This pre-processing method can be used to deal with steep color change-related objects for surfaces-homogeneous training.Furthermore,we develop a multi-level aggregation super resolution photometric stereo network(MASR-PSN)and a novel deep and shallow fusion maxpooling aggregation framework is designed.The proposed deep and shallow fusion max-pooling aggregation framework can be used to enhance feature representation and p

关 键 词：三维重建 光度立体 表面法向恢复 深度学习 超分辨率 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]