局部全局特征耦合与交叉尺度注意的医学图像融合  

Medical Image Fusion with Local-Global Feature Coupling and Cross-Scale Attention

作　　者：张炯 王丽芳[1] 蔺素珍[1] 秦品乐[1] 米嘉 刘阳[1] ZHANG Jiong;WANG Lifang;LIN Suzhen;QIN Pinle;MI Jia;LIU Yang(Shanxi Provincial Key Laboratory of Biomedical Imaging and Imaging Big Data,College of Big Data,North University of China,Taiyuan 030051,China)

机构地区：[1]中北大学大数据学院山西省生物医学成像与影像大数据重点实验室,太原030051

出　　处：《计算机工程》2023年第3期238-247,共10页Computer Engineering

基　　金：山西省应用基础研究计划“纹理和边缘信息在癌症、脑肿瘤医学影像融合中的探索研究”(201901D111152)。

摘　　要：现有基于深度学习的多模态医学图像融合方法存在全局特征表示能力不足的问题。对此,提出一种基于局部全局特征耦合与交叉尺度注意的医学图像融合方法。该方法由编码器、融合规则和解码器三部分组成。编码器中采用并行的卷积神经网络(CNN)和Transformer双分支网络分别提取图像的局部特征与全局表示。在不同尺度下,通过特征耦合模块将CNN分支的局部特征嵌入Transformer分支的全局特征表示中,最大程度地结合互补特征,同时引入交叉尺度注意模块实现对多尺度特征表示的有效利用。编码器提取待融合原始图像的局部、全局以及多尺度特征表示,根据融合规则融合不同源图像的特征表示后再输入到解码器中生成融合图像。实验结果表明,与CBF、PAPCNN、IFCNN、DenseFuse和U2Fusion方法相比,该方法在特征互信息、空间频率、边缘信息传递因子、结构相似度、感知图像融合质量这5个评价指标上分别平均提高6.29%、3.58%、29.01%、5.34%、5.77%,融合图像保留了更清晰的纹理细节和更高的对比度,便于疾病的诊断与治疗。To address the insufficient global feature representation in existing deep learning-based multimodal medical image fusion methods,this study proposes a medical image fusion method based on local-global feature coupling and cross-scale attention.The method comprises three parts:encoder,fusion rule,and decoder.In the encoder,parallel Convolutional Neural Network(CNN)and Transformer dual-branch networks are used to extract the local features and global representation of the image,respectively.At different scales,the local features of the CNN branch are embedded into the global feature representation of the Transformer branch through the feature coupling module for combining complementary features;simultaneously,a cross-scale attention module is introduced to effectively utilize multiscale feature representation.The encoder extracts the local,global,and multiscale feature representations of the original images to be fused,fuses the feature representations of different source images through fusion rules,and then inputs them into the decoder to generate the fused image.Experiments show that compared with CBF,PAPCNN,IFCNN,DenseFuse,and U2Fusion methods,the proposed method objectively improves the five evaluation indicators of feature mutual information,spatial frequency,edge information transfer factor,structural similarity,and perceptual image fusion quality by 6.29%,3.58%,29.01%,5.34%,and 5.77%,respectively;subjectively,the fusion images obtained using this method retain clearer texture details and higher contrast,which is convenient for disease diagnosis and treatment.

关 键 词：医学图像融合 编码器-解码器网络 Transformer网络 特征耦合 交叉尺度注意 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]