基于自注意力模态融合网络的跨模态行人再识别方法研究  

Self-attention Cross-modality Fusion Network for Cross-modality Person Re-identification

作　　者：杜鹏 宋永红[1,2] 张鑫瑶 DU Peng;SONG Yong-Hong;ZHANG Xin-Yao(School of Software Engineering,Xi'an Jiaotong University,Xi'an 710049;College of Artificial Intelligence,Xi'an Jiaotong University,Xi'an 710049)

机构地区：[1]西安交通大学软件学院,西安710049 [2]西安交通大学人工智能学院,西安710049

出　　处：《自动化学报》2022年第6期1457-1468,共12页Acta Automatica Sinica

基　　金：国家重点研究发展计划(2017YFB1301101);陕西省自然科学基础研究计划(2018JM6104)资助。

摘　　要：行人再识别是实现多目标跨摄像头跟踪的核心技术,该技术能够广泛应用于安防、智能视频监控、刑事侦查等领域.一般的行人再识别问题面临的挑战包括摄像机的低分辨率、行人姿态变化、光照变化、行人检测误差、遮挡等.跨模态行人再识别相比于一般的行人再识别问题增加了相同行人不同模态的变化.针对跨模态行人再识别中存在的模态变化问题,本文提出了一种自注意力模态融合网络.首先是利用CycleGAN生成跨模态图像.在得到了跨模态图像后利用跨模态学习网络同时学习两种模态图像特征,对于原始数据集中的图像利用SoftMax损失进行有监督的训练,对生成的跨模态图像利用LSR (Label smooth regularization)损失进行有监督的训练.之后,使用自注意力模块将原始图像和CycleGAN生成的图像进行区分,自动地对跨模态学习网络的特征在通道层面进行筛选.最后利用模态融合模块将两种筛选后的特征进行融合.通过在跨模态数据集SYSU-MM01上的实验证明了本文提出的方法和跨模态行人再识别其他方法相比有一定程度的性能提升.Person re-identification is the core technology to achieve multi-target multi-camera tracking. It can be widely used in many areas such as security, intelligent video surveillance, and criminal investigation. Person re-identification is a challenging task due to the low resolution of camera, human pose variations, illumination variations,pedestrian detector errors and occlusion. Compared with the general person re-identification, the cross-modality person re-identification has the variations of different modalities of the same person. In order to solve the cross-modality problem in cross-modality person re-identification, we propose the self-attention cross-modality fusion network.First, CycleGAN is used to generate cross-modality images. After obtaining the cross-modality images, we use the cross-modality learning network to learn the two modalities features simultaneously. SoftMax loss is used to train original images and label smooth regularization(LSR) loss is used to train generated images. Then, we use self-attention module to distinguish between original images and the generated image, and automatically select the useful features between channels. Finally, modality fusion module is used to fuse these selected features from two modalities images. Comparing with state-of-the-art methods on a large scale cross-modality dataset SYSU-MM01 further demonstrate the effectiveness of the proposed self-attention cross-modality fusion network.

关 键 词：跨模态行人再识别 自注意力 跨模态融合 CycleGAN 

分 类 号：TP391.41[自动化与计算机技术—计算机应用技术]