基于彩色图像高频信息引导的深度图超分辨率重建算法研究  

Study on Algorithm of Depth Image Super-resolution Guided by High-frequency Information of Color Images

作　　者：李嘉莹 梁宇栋 李少吉 张昆鹏 张超[1,2] LI Jiaying;LIANG Yudong;LI Shaoji;ZHANG Kunpeng;ZHANG Chao(School of Computer and Information Technology,Shanxi University,Taiyuan 030006,China;Key Laboratory of Ministry of Education for Computation Intelligence and Chinese Information Processing,Shanxi University,Taiyuan 030006,China)

机构地区：[1]山西大学计算机与信息技术学院,太原030006 [2]山西大学计算智能与中文信息处理教育部重点实验室,太原030006

出　　处：《计算机科学》2024年第7期197-205,共9页Computer Science

基　　金：国家自然科学基金(61802237,62272284);山西省基础研究计划项目(202203021221002,202203021211291);山西省自然科学基金(201901D211176,202103021223464);山西省高等学校科技创新项目(2019L0066);山西省科技重大专项计划(202101020101019);山西省重点研发计划(202102070301019);山西省科技创新青年人才团队项目(202204051001015)。

摘　　要：深度图像信息是三维场景信息的重要组成部分,然而,由于采集设备的局限性和成像环境的多样性,深度传感器获取的深度图像往往分辨率较低、高频信息较少,限制了其在各种计算机视觉任务中的进一步应用。深度图超分辨率试图提高深度图的分辨率,是一项实用而有价值的任务。同一场景下的RGB图像分辨率高,纹理信息丰富,部分深度图超分辨率算法通过引入来自同一场景下的RGB图像提供指导信息,实现了算法性能的显著提升。然而,由于RGB图像和深度图之间的模态不一致,如何充分、有效地利用RGB信息辅助深度图像进行图像超分辨率重建仍然极具挑战。为此,提出了一种基于彩色图像高频信息引导的深度图超分辨率重建算法。具体地,设计了一个高频特征提取模块来自适应地学习彩色图像中的高频信息,以指导深度图边缘的重建。另外,设计了一个特征自注意力模块来获取特征之间的全局依赖,同时提取更深层次的特征,以帮助深度图细节信息的恢复。经过跨模态融合,重组深度图像特征和彩色图像引导特征,并使用多尺度特征融合模块融合不同尺度特征之间的空间结构信息,获取包含多级感受野的重建信息。最后,通过深度重建模块,恢复相应的高分辨率深度图。公开数据集上的实验结果表明所提方法在定量和定性两方面均优于对比方法,验证了所提方法的有效性。Depth image information is an important part of 3D scene information.However,due to the limitations of acquisition equipment and the diversity of imaging environments,the depth images acquired by depth sensors often have low resolution and less high-frequency information,which limits their further applications in various computer vision tasks.Depth image super-resolution attempts to improve the resolution of depth images and is a practical and valuable task.The RGB image in the same scene has high resolution and rich texture information,and some depth image super-resolution algorithms achieve significant improvement in algorithm performance by introducing RGB images from the same scene to provide guidance information.However,due to the structural inconsistency between RGB images and depth maps,how to utilize RGB information fully and effectively is still extremely challenging.To this end,this paper proposes a depth image super-resolution guided by high-frequency information of co-lor images.Specifically,a high-frequency feature extraction module is designed to adaptively learn high-frequency information of color images to guide the reconstruction of depth map edges.In addition,a feature self-attention module is designed to capture the global dependencies between features,extract deeper features to help recover details in the depth image.After cross-modal fusion,the depth image features and color image-guided features are reconstructed,and the proposed multi-scale feature fusion module is used to fuse the spatial structure information between different scale features to obtain reconstruction information including multi-level receptive fields.Finally,through the depth reconstruction module,the corresponding high-resolution depth map is recovered.Comprehensive qualitative and quantitative experimental results on public datasets have demonstrated that the proposed method outperforms comparative methods,which verifies its effectiveness.

关 键 词：深度图超分重建 深度学习 跨模态特征融合 高频信息 自注意力机制 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]