Self-Aligning Multi-Modal Transformer for Oropharyngeal Swab Point Localization  

作　　者：Tianyu Liu Fuchun Sun 

机构地区：[1]Department of Computer Science and Technology,Tsinghua University,Beijing 100083,China

出　　处：《Tsinghua Science and Technology》2024年第4期1082-1091,共10页清华大学学报自然科学版（英文版）

基　　金：supported in part by the Sino-German Collaborative Research Project Crossmodal Learning(No.NSFC 62061136001/DFG TRR169).

摘　　要：The oropharyngeal swabbing is a pre-diagnostic procedure used to test various respiratory diseases, including COVID and Influenza A (H1N1). To improve the testing efficiency of testing, a real-time, accurate, and robust sampling point localization algorithm is needed for robots. However, current solutions rely heavily on visual input, which is not reliable enough for large-scale deployment. The transformer has significantly improved the performance of image-related tasks and challenged the dominance of traditional convolutional neural networks (CNNs) in the image field. Inspired by its success, we propose a novel self-aligning multi-modal transformer (SAMMT) to dynamically attend to different parts of unaligned feature maps, preventing information loss caused by perspective disparity and simplifying overall implementation. Unlike preexisting multi-modal transformers, our attention mechanism works in image space instead of embedding space, rendering the need for the sensor registration process obsolete. To facilitate the multi-modal task, we collected and annotate an oropharynx localization/segmentation dataset by trained medical personnel. This dataset is open-sourced and can be used for future multi-modal research. Our experiments show that our model improves the performance of the localization task by 4.2% compared to the pure visual model, and reduces the pixel-wise error rate of the segmentation task by 16.7% compared to the CNN baseline.

关 键 词：segmentation LOCALIZATION TRANSFORMER multi-modal perception robotic perception 

分 类 号：TP242[自动化与计算机技术—检测技术与自动化装置] TP181[自动化与计算机技术—控制科学与工程]