多模态焊缝图像像素级配准技术  

Pixel-level registration technology for multi-modal weld seam images

作　　者：黄振荣 黄瑶 涂旺 王飞 陈斌 HUANG Zhenrong;HUANG Yao;TU Wang;WANG Fei;CHEN Bin(International Institute of Artificial Intelligence,Harbin Institute of Technology,Shenzhen,Shenzhen Guangdong 518055,China;Shanghai Spaceflight Precision Machinery Institute,Shanghai Academy of Spaceflight Technology,Shanghai 201600,China;Chongqing Research Institute,Harbin Institute of Technology,Chongqing 401151,China)

机构地区：[1]哈尔滨工业大学(深圳)国际人工智能研究院,广东深圳518055 [2]上海航天技术研究院上海航天精密机械研究所,上海201600 [3]哈尔滨工业大学重庆研究院,重庆401151

出　　处：《计算机应用》2024年第S2期229-233,共5页journal of Computer Applications

基　　金：深圳市2022年高校稳定支持计划项目(GXWD-20220811170603002)。

摘　　要：为了挖掘多模态工业焊缝图像视觉特征并通过模态翻译的方式进一步提高多模态工业图像的配准效果,提出一种基于模态翻译的多模态焊缝图像像素级配准网络。首先,通过设计跨模态翻译模块赋予网络捕获不同模态工业图像共享特征的感知能力;其次,捕获共享特征以进行多模态图像配准,并利用对抗性损失和多层级对比损失提高模态翻译效果;同时,结合跨模态翻译模块与单模态图像配准模块,并通过重构损失提升像素级配准性能;最后,构建多模态工业焊缝图像数据集,并基于此数据集开展对比实验。实验结果表明,相较于DFMIR(Discriminator-Free-Medical-Image-Registration)和IMSE(Indescribable Multi-modal Spatial Evaluator)等现有的先进多模态图像配准模型,所提网络在平均交并比(mIoU)上提升了3.9、3.2个百分点,在平均欧氏距离(aEd)上提升了约16、11个像素点的配准精度,在像素级别配准上取得了较好的结果。To exploit the visual features of multi-modal industrial weld seam images and further improve the registration effect through modal translation,a modal translation-based network for pixel-level registration of multi-modal weld seam images was proposed.Firstly,a cross-modal translation module was designed to make the network have the capability to capture shared features of different modalities of industrial images.Then,the shared features were captured to perform multi-modal image registration.At the same time,adversarial loss and multi-level contrastive loss were used to improve the modal translation effect.Additionally,the cross-modal translation module was integrated with the unimodal image registration module,and reconstruction loss was employed to improve pixel-level registration performance.Finally,a multi-modal industrial weld seam image dataset was constructed,and experiments were conducted using this dataset for comparison.Experimental results demonstrate that the proposed network significantly outperforms the existing advanced multi-modal image registration models such as DFMIR(Discriminator-Free-Medical-Image-Registration)and IMSE(Indescribable Multi-modal Spatial Evaluator),achieving 3.9 and 3.2 percentage point increases in mean Intersection over Union(mIoU)and 16-and 11-pixel registration accuracy improvements in average Euclidean distance(aEd),thereby obtaining good results in pixel-level registration.

关 键 词：工业焊缝成像 多模态图像配准 模态翻译 共享特征 多层级对比学习 

分 类 号：TP391.4[自动化与计算机技术—计算机应用技术]