基于双分支U形Transformer的遥感图像融合  

Remote Sensing Image Fusion Based on Two-branch U-shaped Transformer

作　　者：范文盛 刘帆 李明 FAN Wensheng;LIU Fan;LI Ming(College of Data Science,Taiyuan University of Technology,Jinzhong 030600,China)

机构地区：[1]太原理工大学大数据学院,晋中030600

出　　处：《光子学报》2023年第4期179-193,共15页Acta Photonica Sinica

基　　金：国家自然科学基金(No.61703299)。

摘　　要：为解决现有遥感图像融合方法不能充分提取和利用全局上下文特征,而造成光谱和空间信息丢失的问题,提出基于双分支U形Transformer的遥感图像融合方法。首先将多光谱和全色图像分割成图像块,每个图像块的光谱和空间信息被嵌入到一个向量中,形成块嵌入向量序列。接着,多光谱图像和全色图像的嵌入向量序列被分别送入Transformer编码器的两个分支以提取两张图像的多级全局特征表示。在编码过程中,通过多个跳跃连接,不同层级的全色图像表示被注入到多光谱图像表示中进行融合。最终,Transformer解码器利用特征压缩和块扩张层消除冗余特征并从多级融合表示中逐渐恢复出高分辨率融合图像。在三种不同卫星数据集上的实验表明,所提方法得到的融合图像相比于其他融合方法主观视觉效果较好,且客观评价指标更优。Multi-spectral images are key references for earth observation.However,capturing rich spectral information introduces limited spatial resolution in multi-spectral imaging.To overcome the trade-off between spatial resolution and spectral resolution in remote sensing,panchromatic images with high spatial resolution but poor spectral information are adopted to complement multi-spectral imagery.As a result,the technique of fusing high-resolution panchromatic images and low-resolution multi-spectral images,namely pan-sharpening,is developed and facilitates various remote sensing application.Existing pan-sharpening methods can be roughly divided into four main categories:component substitution,multi-resolution analysis,variational optimization and deep learning.Each category has its own fusion strategy.Recently,a number of deep-learning-based methods are developed and obtain superior performance on fusion quality.These methods are typically based on convolutional neural networks and even combine the idea of generative adversarial networks.However,the inadequate extraction of global contextual and multi-scale features always leads to a loss of spectral information and spatial details.To solve this problem,a twobranch u-shaped transformer is proposed in this paper.Firstly,the multi-spectral and panchromatic images to be fused are partitioned into non-overlapping patches with fixed patch sizes,and each patch is embedded into a vector.The embedding vectors have the same feature dimension and contain the rich spectral and spatial information of the image patches.Subsequently,the embedding vectors of the multi-spectral and panchromatic images are fed into the two branches of the transformer encoder to extract hierarchical feature representations,respectively.The encoder consists of shifted windowing transformer blocks and patch emerging layers.Therefore,it can fully extract global and multi-scale features.In the encoding process,hierarchical panchromatic feature representations are injected into multi-spectral feature repres

关 键 词：遥感图像融合 深度学习 TRANSFORMER 多光谱图像 全色图像 

分 类 号：TP751[自动化与计算机技术—检测技术与自动化装置]