带深度信息监督的神经辐射场虚拟视点画面合成  

Virtual viewpoint image synthesis using neural radiance fields with depth information supervision

作　　者：刘晓楠 陈纯毅[1] 胡小娟[1] 于海洋[1] Liu Xiaonan;Chen Chunyi;Hu Xiaojuan;Yu Haiyang(School of Computer Science and Technology,Changchun University of Science and Technology,Changchun 130022,China)

机构地区：[1]长春理工大学计算机科学技术学院,长春130022

出　　处：《中国图象图形学报》2024年第7期2035-2045,共11页Journal of Image and Graphics

基　　金：国家自然科学基金项目(U19A2063);吉林省科技发展计划项目(20230201080GX)。

摘　　要：目的 在神经辐射场虚拟视点画面合成过程中,因视图数量过少或视图颜色不一致产生离群稀疏深度值问题,提出利用深度估计网络的密集深度值监督神经辐射场虚拟视点画面合成的方法来解决此问题。方法 首先输入视图进行运动恢复结构获取稀疏深度值,其次将RGB视图输入New CRFs(neural window fully-connected CRFs for monocular depth estimation)深度估计网络得到预估深度值,计算预估深度值和稀疏深度值之间的标准差。最后,利用预估深度值和计算得到的标准差,对神经辐射场的训练进行监督。结果 实验在NeRF Real数据集上与其他算法进行了实验对比。在少量视图合成实验中,本文方法在图像质量和效果优于仅使用RGB监督的NeRF(neural radiance fields)方法和使用稀疏深度信息监督的方法,峰值信噪比较NeRF方法提高24%,较使用稀疏深度信息监督的方法提高19.8%;结构相似度比NeRF方法提高36%,比使用稀疏深度信息监督的方法提高16.6%。同时为了验证算法的数据效率,进行了相同的迭代次数达到的峰值信噪比的比较,相较于NeRF方法,数据效率也有明显提高。结论 实验结果表明,本文所提出的利用深度估计网络密集深度值监督神经辐射场虚拟视点画面合成的方法,解决了视图数量过少或者视图颜色不一致产生离群稀疏深度值问题。Objective Viewpoint synthesis techniques are widely applied to computer graphics and computer vision.In accordance with whether they depend on geometric information or not,virtual viewpoint synthesis methods can be classified into two distinct categories:image-based rendering and model-based rendering.1) Image-based rendering typically utilizes input data from camera arrays or light field cameras to achieve higher-quality rendering outcomes without the need to reconstruct the geometric information of the scene.Among the image-based rendering methods,depth map-based rendering technology is currently a popular research topic for virtual viewpoint rendering.However,this technology is prone to be affected by depth errors,leading to challenges such as holes and artifacts in the generated virtual viewport image.In addition,obtaining precise depth information for real-world scenes poses difficulties in practical applications.2) Model-based rendering involves 3D geometric modeling of real-world scenes.This method utilizes techniques such as projection transformation,cropping,fading,and texture mapping to synthesize virtual viewpoint images.However,quickly modeling realworld scenes is a significant disadvantage of this approach.With the emergence of neural rendering technology,the neural radiance fields technique employs a neural network to represent the 3D scene and combines it with volume rendering technology for viewpoint synthesis,thus producing photo-realistic viewpoint synthesis results.However,this approach is heavily reliant on the appearance of the view and requires a substantial number of views to be input for modeling.As a result,this method may be capable of perfectly explaining the training images but generalizes poorly to novel test views.Depth information is introduced for supervision to reduce the dependence of the neural radiance fields on the view appearance.However,structure from motion produces sparse depth values with inaccuracy and outliers due to the limited number of view inputs.Therefore,this stud

关 键 词：视点合成 神经辐射场(NeRF) 深度监督 深度估计 体渲染 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]