基于多模态传感器的近红外与可见光图像自适应融合模型  

Adaptive Fusion Model for Near-infrared and Visible Light Images Based on Multimodal Sensors

作　　者：李振伟 施文灶 付强[2] 苑俊茹 LI Zhenwei;SHI Wenzao;FU Qiang;YUAN Junru(College of Photonic and Electronic Engineering,Fujian Normal University,Fuzhou 350117,China;Tucsen Photonics Co.,Ltd.,Fuzhou 350003,China;Fujian Provincial Engineering Technology Research Center of Photoelectric Sensing Application,Fujian Normal University,Fuzhou 350117,China;Key Laboratory of Optoelectronic Science and Technology for Medicine(Ministry of Education),Fujian Normal University,Fuzhou 350117,China;Fujian Provincial Key Laboratory for Photonics Technology,Fujian Normal University,Fuzhou 350117,China)

机构地区：[1]福建师范大学光电与信息工程学院,福建福州350117 [2]福建鑫图光电有限公司,福建福州350003 [3]福建师范大学福建省光电传感应用工程技术研究中心,福建福州350117 [4]福建师范大学医学光电科学与技术教育部重点实验室,福建福州350117 [5]福建师范大学福建省光子技术重点实验室,福建福州350117

出　　处：《现代信息科技》2024年第24期163-170,共8页Modern Information Technology

摘　　要：针对现有的图像融合方法在特征提取和融合策略上的不足,提出了一种基于频域分解的近红外与可见光图像自适应融合模型STAFuse。通过引入Transformer与CNN的特征提取模块,以及自适应融合模块,实现不同模态图像特征的有效融合。在多模态图像的获取上,为解决传统多传感器系统体积大、校准复杂等问题,设计了一种新型多模态传感器,可同时获取高分辨率的可见光图像和低分辨率的近红外图像。实验结果表明,STAFuse在多个指标上优于现有模型,在结构相似性上比DenseFuse模型提升了102.7%,在视觉信息保真度上比DIDFuse模型提升了25%,在保持视觉质量和图像细节方面表现突出。Aiming at the shortcomings of feature extraction and fusion strategies in the existing image fusion methods,this paper proposes an adaptive fusion model for near-infrared and visible light images,called STAFuse,based on frequency domain decomposition.It realizes the effective fusion of different modal image features,by introducing feature extraction modules of Transformer and CNN and the adaptive fusion modules.To address the issues of large size and complex calibration in traditional multi-sensor systems on the acquisition of the multimodal images,a novel multimodal sensor is designed,capable of simultaneously capturing high-resolution visible light images and low-resolution near-infrared images.Experimental results demonstrate that STAFuse outperforms existing models in multiple metrics,which improves by 102.7%compared with DenseFuse model in Structural Similarity(SSIM),improves by 25%compared with DIDFuse model in Visual Information Fidelity(VIF),and is outstanding in maintaining visual quality and image details.

关 键 词：近红外与可见光融合 自适应融合 TRANSFORMER CNN 多模态传感器 频域分解 

分 类 号：TP212[自动化与计算机技术—检测技术与自动化装置] TP183[自动化与计算机技术—控制科学与工程]