基于CNN和ViT的三支路生成对抗网络的多源遥感图像融合  

Multi-source Remote Sensing Image Fusion Based on Triple-branch Generative Adversarial Network Constructed by CNN and ViT

作　　者：姬云翔 康家银 马寒雁 章洋洋 张文慧 王怀友 JI Yunxiang;KANG Jiayin;MA Hanyan;ZHANG Yangyang;ZHANG Wenhui;WANG Huaiyou(School of Electronic Engineering,Jiangsu Ocean University,Lianyungang 222005,China)

机构地区：[1]江苏海洋大学电子工程学院,江苏连云港222005

出　　处：《江苏海洋大学学报（自然科学版）》2024年第3期70-76,共7页Journal of Jiangsu Ocean University:Natural Science Edition

基　　金：国家自然科学基金面上资助项目(62271236);江苏海洋大学自然科学基金项目(Z2015009);江苏海洋大学研究生科研与实践创新计划项目(KYCX2022-41)。

摘　　要：多源遥感图像具有信息互补性,其中同一场景的全色图像具有较高空间分辨率,而多光谱图像具有较高光谱分辨率。通过多模态图像融合技术,可将全色图像与多光谱图像的信息进行整合,从而得到同时具有高空间分辨率与高光谱分辨率的融合图像。为此,提出了一种基于CNN和ViT的三支路生成对抗网络的多源遥感图像融合方法。具体地,先将全色图像与多光谱图像分别输入至生成器的空间支路与光谱支路进行特征提取,与此同时将全色图像与多光谱图像经联结后输入至生成器的融合支路进行特征提取;然后,在融合支路,将空间支路、光谱支路提取到的特征逐级地与融合支路提取到的特征进行交互和联结,并重构得到融合图像;接着,采用两个判别器,即空间判别器与光谱判别器,分别对融合图像从空间信息与光谱信息两方面进行真伪判别;最后,通过生成器与两个判别器之间的对抗训练,最终得到同时具有高空间分辨率与高光谱分辨率的融合图像。实验结果表明,相较于CNMF,PanNet,Pan-GAN,SDPNet方法,所提方法得到的融合结果在定性、定量对比方面均具有优越性。Multi-source remote sensing images have complementary information,where panchromatic images of the same scene have higher spatial resolution,while multispectral images have higher spectral resolution.Through multimodal image fusion technology,the information of panchromatic images and multispectral images can be integrated to obtain a fused image with both high spatial resolution and high spectral resolution.To this end,this paper proposes a multi-source remote sensing image fusion method using a triple-branch generative adversarial network constructed by CNN and ViT.Specifically,the panchromatic image and multispectral image are firstly input into the spatial and spectral branches of the generator for feature extraction,respectively.At the same time,the panchromatic image and multispectral image are connected and input into the fusion branch of the generator for feature extraction;Then,in the fusion branch,the features extracted from the spatial and spectral branches are gradually interacted and connected with the features extracted from the fusion branch,and the fused image is reconstructed.Next,two discriminators,namely spatial discriminator and spectral discriminator,are used to distinguish the authenticity of the fused image from both spatial and spectral information aspects.Finally,through adversarial training between the generator and two discriminators,a fused image with both high spatial resolution and high spectral resolution is obtained.Experimental results show that compared with CNMF,PanNet,Pan-GAN,SDPNet methods,the fusion results obtained by the proposed method are superior in both qualitative evaluations and quantitative assessments.

关 键 词：遥感图像融合 卷积神经网络 视觉Transformer 生成对抗网络 

分 类 号：TP18[自动化与计算机技术—控制理论与控制工程]