多模态数据特征融合的广角图像生成  

Multi-modality feature fusion-based wide field-of-view image generation

作　　者：姜智颖 张曾翕 刘晋源 刘日升[2] Jiang Zhiying;Zhang Zengxi;Liu Jinyuan;Liu Risheng(College of Information Science and Technology,Dalian Maritime University,Dalian 116026,China;School of Software Technology&DUT-RU International School of Information Science and Engineering,Dalian University of Technology,Dalian 116081,China;School of Mechanical Engineering,Dalian University of Technology,Dalian 116081,China)

机构地区：[1]大连海事大学信息科学技术学院,大连116026 [2]大连理工大学软件学院、大连理工大学—立命馆大学国际信息与软件学院,大连116081 [3]大连理工大学机械工程学院,大连116081

出　　处：《中国图象图形学报》2025年第1期173-187,共15页Journal of Image and Graphics

基　　金：国家重点研发计划资助(2022YFA1004101);国家自然科学基金项目(62302078,U22B2052);中国博士后科学基金项目(2023M730741)。

摘　　要：目的图像拼接通过整合不同视角的可见光数据获得广角合成图。不利的天气因素会使采集到的可见光数据退化,导致拼接效果不佳。红外传感器通过热辐射成像,在不利的条件下也能突出目标,克服环境和人为因素的影响。方法考虑到红外传感器和可见光传感器的成像互补性,本文提出了一个基于多模态数据(红外和可见光数据)特征融合的图像拼接算法。首先利用红外数据准确的结构特征和可见光数据丰富的纹理细节由粗到细地进行偏移估计,并通过非参数化的直接线性变换得到变形矩阵。然后将拼接后的红外和可见光数据进行融合,丰富了场景感知信息。结果本文选择包含530对可拼接多模态图像的真实数据集以及包含200对合成数据集作为测试数据,选取了3个最新的融合方法,包括RFN(residual fusion network)、ReCoNet(recurrent correction network)和DATFuse(dual attention transformer),以及7个拼接方法,包括APAP(as projective as possible)、SPW(single-perspective warps)、WPIS(wide parallax image stitching)、SLAS(seam-guided local alignment and stitching)、VFIS(view-free image stitching)、RSFI(reconstructing stitched features to images)和UDIS++(unsupervised deep image stitching)组成的21种融合—拼接策略进行了定性和定量的性能对比。在拼接性能上,本文方法实现了准确的跨视角场景对齐,平均角点误差降低了53%,避免了鬼影的出现;在多模态互补信息整合方面,本文方法能自适应兼顾红外图像的结构信息以及可见光图像的丰富纹理细节,信息熵较DATFuse-UDIS++策略提升了24.6%。结论本文方法在结合了红外和可见光图像成像互补优势的基础上,通过多尺度递归估计实现了更加准确的大视角场景生成;与常规可见光图像拼接相比鲁棒性更强。Objective Image stitching,a cornerstone in the field of computer vision,is dedicated to assembling a compre⁃hensive field-of-view image by merging visible data captured from multiple vantage points within a specific scene.This fusion enhances scene perception and facilitates advanced processing.The current state-of-the-art in image stitching pri⁃marily hinges on the detection of feature points within the scene,necessitating their dense and uniform distribution through⁃out the image.However,these approaches encounter significant challenges in outdoor environments or when applied to military equipment,where adverse weather conditions such as rain,haze,and low light can severely degrade the quality of visible images.This degradation impedes the extraction of feature points,a critical step in the stitching process.Further⁃more,factors such as camouflage and occlusion can lead to data loss,disrupting the distribution of feature points and thus compromising the quality of the stitched image.These limitations often manifest as ghosting effects,undermining the effec⁃tiveness of the stitching and its robustness in practical applications.In this challenging context,infrared sensors,which detect thermal radiation to image scenes,emerge as a robust alternative.They excel in highlighting targets even under unfa⁃vorable conditions,mitigating the impact of environmental and human factors.This capability makes them highly valuable in military surveillance applications.However,a significant drawback of thermal imaging is its inability to capture the rich texture details that are abundant in visible images.These details are crucial for an accurate and comprehensive perception of the scene.Method This paper proposes a groundbreaking image stitching algorithm to overcome the limitations inherent in conventional visible image stitching and to extend the applicability of stitching technology across various environments.This algorithm is based on the fusion of features from multi-modality images,specifically,infrared and vi

关 键 词：多模态图像融合 图像拼接 卷积神经网络(CNN) 红外—可见光图像 多尺度金字塔 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]