三维场景注视点渲染深度学习方法综述  

Deep learning-based foveated rendering in 3D space:a review

作　　者：李英群 胡啸 徐翔 徐延宁[1] 王璐 Li Yingqun;Hu Xiao;Xu Xiang;Xu Yanning;Wang Lu(School of Software,Shandong University,Jinan 250101,China;Shandong Key Laboratory of Blockchain Finance,Shandong University of Finance and Economics,Jinan 250014,China)

机构地区：[1]山东大学软件学院,济南250101 [2]山东财经大学山东省区块链金融重点实验室,济南250014

出　　处：《中国图象图形学报》2024年第10期2955-2978,共24页Journal of Image and Graphics

基　　金：国家重点研发计划资助(2022YFB3303203);国家自然科学基金项目(62272275)。

摘　　要：在大型高分辨显示器和头戴式显式设备中实现实时、逼真的渲染仍然是计算机图形学面临的主要挑战之一。注视点渲染(foveated rendering)利用人类视觉系统的局限性,根据注视点调整图像渲染质量,从而在不损失用户感知质量的前提下大大提高渲染速度。随着深度学习方法在渲染领域的广泛应用,涌现出大量基于深度学习的注视点渲染新方法。本文从深度学习的角度对注视点渲染领域的最新方法进行综述。首先,概述了人类视觉感知的背景知识。接着,简要介绍了注视点渲染中最具代表性的非深度学习方法,包括自适应分辨率、几何简化、着色简化和硬件实现,并总结了这些方法的优缺点。随后,描述了文中用于评估深度学习不同方法所使用的评估准则,包括常用的注视点渲染图像的评估指标和注视点预测评估指标。接下来,将注视点渲染中的深度学习方法细分为超分辨率、降噪、补全、图像合成、注视点预测和图像应用,对它们进行详细概述和总结。最后,提出了深度学习方法目前面临的问题和挑战。通过对注视点渲染领域的深度学习方法的讨论,可以更详细地展示深度学习在注视点渲染中的研究前景和发展方向,对后续研究人员在选择研究方向和设计网络架构等方面都有一定的参考价值。The widespread adoption of virtual reality(VR)and augmented reality technologies across various sectors,including healthcare,education,military,and entertainment,has propelled head-mounted displays with high resolution and wide fields of view into the forefront of display devices.However,attaining a satisfactory level of immersion and interactivity poses a primary challenge in the realm of VR,with latency potentially leading to user discomfort in the form of dizziness and nausea.Multiple studies have underscored the necessity of achieving a highly realistic VR experience while maintaining user comfort,entailing the elevation of the screen’s image refresh rate to 1800 Hz and keeping latency below 3~40 ms.Achieving real-time,photorealistic rendering at high resolution and low latency represents a formidable objective.Foveated rendering is an effective approach to address these issues by adjusting the rendering quality across the image based on gaze position,maintaining high quality in the fovea area while reducing quality in the periphery.This technique leads to substantial computational savings and improved rendering speed without a perceptible loss in visual quality.While previous reviews have examined technical approaches to foveated rendering,they focused more on categorizing the imple mentation techniques.A comprehensive review within the domain of machine learning still needs to be explored.With the ongoing advancements in machine learning within the rendering field,combining machine learning and foveated rendering is considered a promising research area,especially in postprocessing,where machine learning methods have great potential.Nonmachine learning methods inevitably introduce artifacts.By contrast,machine learning methods have a wide range of applications in the postprocessing domain of rendering to optimize and improve foveated rendering results and enhance the realism and immersion of foveated images in a manner unattainable through nonmachine learning approaches.Therefore,this work presents a compr

关 键 词：注视点渲染 深度学习 实时渲染 注视点预测 图像补全 超分辨率 光路追踪降噪 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]