面向跨模态行人重识别的单模态自监督信息挖掘  

Single-modality self-supervised information mining for cross-modality person re-identification

作　　者：吴岸聪 林城梽 郑伟诗[1] Wu Ancong;Lin Chengzhi;Zheng Weishi(School of Computer Science and Engineering,Sun Yat-sen University,Guangzhou 510006,China)

机构地区：[1]中山大学计算机学院,广州510006

出　　处：《中国图象图形学报》2022年第10期2843-2859,共17页Journal of Image and Graphics

基　　金：国家自然科学基金青年科学基金项目(62106288);中国博士后创新人才支持计划(BX20200395);中国博士后科学基金面上资助(2021M693616)。

摘　　要：目的在智能监控视频分析领域中,行人重识别是跨无交叠视域的摄像头匹配行人的基础问题。在可见光图像的单模态匹配问题上,现有方法在公开标准数据集上已取得优良的性能。然而,在跨正常光照与低照度场景进行行人重识别的时候,使用可见光图像和红外图像进行跨模态匹配的效果仍不理想。研究的难点主要有两方面:1)在不同光谱范围成像的可见光图像与红外图像之间显著的视觉差异导致模态鸿沟难以消除;2)人工难以分辨跨模态图像的行人身份导致标注数据缺乏。针对以上两个问题,本文研究如何利用易于获得的有标注可见光图像辅助数据进行单模态自监督信息的挖掘,从而提供先验知识引导跨模态匹配模型的学习。方法提出一种随机单通道掩膜的数据增强方法,对输入可见光图像的3个通道使用掩膜随机保留单通道的信息,使模型关注提取对光谱范围不敏感的特征。提出一种基于三通道与单通道双模型互学习的预训练与微调方法,利用三通道数据与单通道数据之间的关系挖掘与迁移鲁棒的跨光谱自监督信息,提高跨模态匹配模型的匹配能力。结果跨模态行人重识别的实验在“可见光—红外”多模态行人数据集SYSU-MM01(Sun Yat-Sen University Multiple Modality 01)、RGBNT201(RGB,near infrared,thermal infrared,201)和RegDB上进行。实验结果表明,本文方法在这3个数据集上都达到领先水平。与对比方法中的最优结果相比,在RGBNT201数据集上的平均精度均值mAP(mean average precision)有最高接近5%的提升。结论提出的单模态跨光谱自监督信息挖掘方法,利用单模态可见光图像辅助数据挖掘对光谱范围变化不敏感的自监督信息,引导单模态预训练与多模态有监督微调,提高跨模态行人重识别的性能。Objective Urban video surveillance systems have been developing dramatically nowadays.The surveillance videos analysis is essential for security but a huge amount of labor-intensive data processing is highly time-consuming and costly.Intelligent video analysis can be as an effective way to deal with that.To analyze the concrete pedestrians’event,person re-identification is a basic issue of matching pedestrians across non-overlapping cameras views for obtaining the trajectories of persons in a camera network.The cross-camera scene variations are the key challenges for person re-identification,such as illumination,resolution,occlusions and background clutters.Thanks to the development of deep learning,single-modality visible image matching has achieved remarkable performance on benchmark datasets.However,visible image matching is not applicable in low-light scenarios like night-time outdoor scenes or dark indoor scenes.To resilient the related lowlight issues,most of surveillance cameras can automatically switch to acquire near infrared images,which are visually different from visible images.When person re-identification is required for the penetration between normal-light and low-light,current person re-identification performance for cross-modality matching between visible images and infrared images cannot be satisfied.Thus,it is necessary to analyze the visible-infrared cross-modality person re-identification further.For visibleinfrared cross-modality person re-identification,there are two key challenges as mentioned below:first,the spectrums and visual appearances of visible images and infrared images are significantly different.Visible images contain three channels of red(R),green(G)and blue(B)responses,while infrared images contain only one channel of near infrared responses.This leads to big modality gap.Next,lack of labeled data is still challenged based on manpower-based identification of the same pedestrian across visible image and infrared image.Current multi-modality benchmark dataset contains 500 pers

关 键 词：行人重识别 跨模态检索 红外图像 自监督学习 互学习 

分 类 号：TP301.6[自动化与计算机技术—计算机系统结构]