基于深度学习的图像融合方法综述  

Deep learning-based image fusion: a survey

作　　者：唐霖峰 张浩[1] 徐涵 马佳义 Tang Linfeng;Zhang Hao;Xu Han;Ma Jiayi(Electronic Information School,Wuhan University,Wuhan 430072,China)

机构地区：[1]武汉大学电子信息学院,武汉430072

出　　处：《中国图象图形学报》2023年第1期3-36,共34页Journal of Image and Graphics

基　　金：国家自然科学基金项目(62276192);湖北省自然科学基金项目(2020BAB113)。

摘　　要：图像融合技术旨在将不同源图像中的互补信息整合到单幅融合图像中以全面表征成像场景,并促进后续的视觉任务。随着深度学习的兴起,基于深度学习的图像融合算法如雨后春笋般涌现,特别是自编码器、生成对抗网络以及Transformer等技术的出现使图像融合性能产生了质的飞跃。本文对不同融合任务场景下的前沿深度融合算法进行全面论述和分析。首先,介绍图像融合的基本概念以及不同融合场景的定义。针对多模图像融合、数字摄影图像融合以及遥感影像融合等不同的融合场景,从网络架构和监督范式等角度全面阐述各类方法的基本思想,并讨论各类方法的特点。其次,总结各类算法的局限性,并给出进一步的改进方向。再次,简要介绍不同融合场景中常用的数据集,并给出各种评估指标的具体定义。对于每一种融合任务,从定性评估、定量评估和运行效率等多角度全面比较其中代表性算法的性能。本文提及的算法、数据集和评估指标已汇总至https://github.com/Linfeng-Tang/Image-Fusion。最后,给出了本文结论以及图像融合研究中存在的一些严峻挑战,并对未来可能的研究方向进行了展望。Image fusion aims to integrate complementary information from multi-source images into a single fused image to characterize the imaging scene and facilitate the subsequent vision tasks further. In recent years, it has been a concern in the field of image processing, especially in artificial intelligence-related industries like intelligent medical service, autonomous driving, smart photography, security surveillance, and military monitoring. Moreover, the growth of deep learning has been promoting deep learning-based image fusion algorithms. In particular, the emergence of advanced techniques, such as the auto-encoder, generative adversarial network, and Transformer, has led to a qualitative leap in image fusion performance. However, a comprehensive review and analysis of state-of-the-art deep learning-based image fusion algorithms for different fusion scenarios are required to be realized. Thus, we develop a systematic and critical review to explore the developments of image fusion in recent years. First, a comprehensive and systematic introduction of the image fusion field is presented from the following three aspects: 1) the development of image fusion technology, 2) the prevailing datasets, and 3) the common evaluation metrics. Then, more extensive qualitative experiments, quantitative experiments, and running efficiency evaluations of representative image fusion methods are conducted on the public datasets to compare their performance. Finally, the summary and challenges in the image fusion community are highlighted. In particular, some prospects are recommended further in the field of image fusion. First of all, from the perspective of fusion scenarios, the existing image fusion methods can be divided into three categories, i.e., multi-modal image fusion, digital photography image fusion, and remote sensing image fusion. Specifically, multi-modal image fusion is composed of infrared and visible image fusion as well as medical image fusion, and digital photography image fusion consists of multi-exposure image

关 键 词：图像融合 深度学习 多模图像 数字摄影 遥感影像 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]