多尺度模态感知在文本指代实例分割中的研究与应用  

Multi-scale modality perception network for referring image segmentation

作　　者：刘静[1] 胡永利[1] 刘秀平[2] 谭红臣 尹宝才[1] LIU Jing;HU Yong-li;LIU Xiu-ping;TAN Hong-chen;YIN Bao-cai(School of Artificial Intelligence and Automation,Beijing University of Technology,Beijing 100124,China;School of Mathematical Sciences,Dalian University of Technology,Dalian Liaoning 116024,China)

机构地区：[1]北京工业大学人工智能与自动化学院,北京100124 [2]大连理工大学数学科学学院,辽宁大连116024

出　　处：《图学学报》2022年第6期1150-1158,共9页Journal of Graphics

基　　金：第7批全国博士后创新人才支持计划(BX20220025);第70批全国博士后面上资助(2021M700303)。

摘　　要：文本指代实例分割(RIS)任务是解析文本描述所指代的实例,并在对应图像中分割出该实例,是计算机视觉与媒体领域中热门的研究课题。当前,大多数RIS方法基于单尺度文本/图像模态信息的融合,以感知指代实例的位置和语义信息。然而,单一尺度模态信息很难同时涵盖定位不同大小实例所需的语义和结构上下文信息,阻碍了模型对任意大小指代实例的感知,进而影响模型对不同大小指代实例的分割。对此,设计多尺度视觉-语言交互感知模块和多尺度掩膜预测模块:前者增强模型对不同尺度实例语义与文本语义之间的融合与感知;后者通过充分捕捉不同尺度实例的所需语义和结构信息提升指代实例分割的表现。由此,提出了多尺度模态感知的文本指代实例分割模型(MMPN-RIS)。实验结果表明,MMPN-RIS模型在RefCOCO,RefCOCO+和RefCOCOg3个公开数据集的oIoU指标上均达到了前沿性能;针对文本指代不同尺度实例的分割,MMPN-RIS模型有着较好的表现。Referring image segmentation(RIS) is the task of parsing the instance referred to by the text description and segmenting the instance in the corresponding image. It is a popular research topic in computer vision and media.Currently, most RIS methods are based on the fusion of single-scale text/image modality information to perceive the location and semantic information of referential instances. However, it is difficult for single-scale modal information to simultaneously cover both the semantics and structural context information required to locate instances of different sizes. This defect hinders the model from perceiving referent instances of any size, which affects the model’s segmentation of referent instances of different sizes. This paper designed a Multi-scale Visual-Language Interaction Perception Module and a Multi-scale Mask Prediction Module to solve this problem. The former could enhance the model’s ability to perceive instances at different scales and promote effective alignment of semantics between different modalities. The latter could improve the performance of referring instance segmentation by fully capturing the required semantic and structural information of instances at different scales. Therefore, this paper proposed a multi-scale modality perception network for referring image segmentation(MMPN-RIS). The experimental results show that the MMPN-RIS model has achieved cutting-edge performance on the oIoU indicators of the three public datasets RefCOCO, RefCOCO+, and RefCOCOg. For the RIS of different scales, the MMPN-RIS model could also yield good performance.

关 键 词：视觉与语言 文本指代实例分割 异模态融合与感知 特征金字塔 

分 类 号：TP391[自动化与计算机技术—计算机应用技术]