Cross modality person re-identification based on residual enhanced attention

doi:10.11996/JG.j.2095-302X.2023010033

Abstract

Abstract:

Cross modality person re-identification mainly faces two problems: ① Modality discrepancies between infrared and visible images caused by different imaging mechanisms. ② Intra-class discrepancies caused by the insufficient identity discrimination of features. In order to address the above two problems, a cross modality person re-identification method based on residual enhanced attention was proposed to improve the modality invariance and identity discrimination of pedestrian features. First, with non-shared parameters at the shallow network and shared parameters at the deep layer a dual-stream convolutional neural network was designed as the backbone. Then, the problem of global weakening in the existing attention mechanism was analyzed, and a residual enhancement method was designed to solve this problem and improve the performance of the attention mechanism. It was applied to the shallow channel dimension and deep spatial location of the network respectively. Sufficient experiments on the two datasets SYSU-MM01 and RegDB have proved the effectiveness of the method.

Key words: person re-identification, residual enhancement, attention mechanism, modality invariance, neural network

CLC Number:

TP391

SHAO Wen-bin, LIU Yu-jie, SUN Xiao-rui, LI Zong-min. Cross modality person re-identification based on residual enhanced attention[J]. Journal of Graphics, 2023, 44(1): 33-40.

Figures/Tables 8

References 29

[1]	YE M, SHEN J, LIN G, et al. Deep learning for person re-identification: a survey and outlook[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6): 2872-2893. DOI URL
[2]	杨文娟, 王文明, 王全玉, 等. 基于感知哈希和视觉词袋模型的图像检索方法[J]. 图学学报, 2019, 40(3): 519-524.
	YANG W J, WANG W M, WANG Q Y, et al. Image retrieval method based on perceptual hash algorithm and bag of visual words[J]. Journal of Graphics, 2019, 40(3): 519-524 (in Chinese).
[3]	ZHAO H, TIAN M, SUN S, et al. Spindle net: person re-identification with human body region guided feature decomposition and fusion[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1077-1085.
[4]	GE Y X, ZHU F, CHEN D P, et al. Self-paced contrastive learning with hybrid memory for domain adaptive object re-id[EB/OL]. [2021-12-08]. https://blog.csdn.net/NGUever15/article/details/120556059.
[5]	CHEN H, WANG Y, LAGADEC B. Joint generative and contrastive learning for unsupervised person re-identification[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 2004-2013.
[6]	WU A, ZHENG W, YU H, et al. RGB-infrared cross-modality person re-identification[C]//2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 5380-5389.
[7]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// The IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7132-7141.
[8]	WOO S, PARK J, LEE J Y, et al. Cbam: convolutional block attention module[C]// 2018 European Conference on Computer Vision. Cham: Springer International Publishin, 2018: 3-19.
[9]	DAI P Y, JI R R, WANG H, WU Q, el al. Cross-modality person re-identification with generative adversarial training[EB/OL]. [2021-12-08]. https://www.ijcai.org/Proceedings/2018/0094.pdf.
[10]	WANG Z X, WANG Z, ZHENG Y. Learning to reduce dual-level discrepancy for infrared-visible person re-identification[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 618-626.
[11]	WANG G, ZHANG T, CHENG J, et al. RGB-infrared cross-modality person re-identification via joint pixel and feature alignment[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 3623-3632.
[12]	ZHAO Z, LIU B, CHU Q, et al. Joint color-irrelevant consistency learning and identity-aware modality adaptation for visible-infrared cross modality person re-identification[C]// The AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2021: 3520-3528.
[13]	YE M, LAN X, WANG Z, et al. Bi-directional center- constrained top-ranking for visible thermal person re-identification[C]//IEEE Transactions on Information Pernsics and Security. New York: IEEE Press, 2020: 407-419.
[14]	HAO Y, WANG N, LI J, el al. Hsme: hypersphere manifold embedding for visible thermal person re-identification[C]//The AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2019: 8385-8392.
[15]	JIA M X, ZHAI Y P, LU S J, et al. A similarity inference metric for RGB-infrared cross-modality person re-identification[EB/OL]. [2021-12-08]. https://arxiv.org/abs/2007.01504
[16]	LI D, WEI X, HONG X, el al. Infrared-visible cross-modal person re-identification with an x modality[C]//The AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2020: 4610-4617.
[17]	YE M, SHEN J, CRANDALL D, et al. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification[C]//2020 European Conference on Computer Vision - ECCV 2020. Cham: Springer International Publishin, 2020: 229-247.
[18]	CHEN Y, WAN L, LI Z, et al. Neural feature search for rgb-infrared person re-identification[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, New York: IEEE Press, 2021: 587-597.
[19]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[EB/OL]. [2021-12-08]. https://blog.csdn.net/toda666/article/details/80384915.
[20]	LUO H, GU Y, LIAO X, et al. Bag of tricks and a strong baseline for deep person re-identification[EB/OL]. [2021-12-08]. https://openaccess.thecvf.com/content_CVPRW_2019/papers/TRMTMCT/Luo_Bag_of_Tricks_and_a_Strong_Baseline_for_Deep_Person_CVPRW_2019_paper.pdf.
[21]	DALAL N, TRIGGS B. Histograms of oriented gradients for human detection[C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2005: 886-893.
[22]	NGUYEN D T, HONG H G, KIM K W, et al. Person recognition system based on a combination of body images from visible light and thermal cameras[J]. Sensors: Basel, Switzerland, 2017, 17(3): E605.
[23]	WANG G, ZHANG T, YANG Y, et al. Cross-modality paired-images generation for RGB-infrared person re-identification[C]// The AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2020: 12144-12151.
[24]	DENG J, DONG W, SOCHER R, et al. Imagenet: a large-scale hierarchical image database[C]// 2009 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2009: 248-255.
[25]	KINGMA D, BA J. Adam: a method for stochastic optimization[EB/OL]. [2021-12-08]. https://arxiv.org/abs/1412.6980.
[26]	YE M, LAN X, LENG Q, et al. Cross-modality person re-identification via modality-aware collaborative ensemble learning[C]// 2020 IEEE Transactions on Image Processing. New York: IEEE Press, 2020: 9387-9399.
[27]	LU Y, WU Y, LIU B, et al. Cross-modality person re-identification with shared-specific feature transfer[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 13376-13386.
[28]	YE M, RUAN W J, DU B, et al. Channel augmented joint learning for visible-infrared recognition[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 13567-13576.
[29]	SELVARAJU R, COGSWELL M, DAS A. Grad-cam: visual explanations from deep networks via gradient-based localization[J]. International Journal of Computer Vision, 2020, 128(2): 336-359. DOI

方法	年份	All-search (%)								Indoor-search (%)
		Single-shot				Multi-shot				Single-shot				Multi-shot
		R-1	R-10	R-20	mAP	R-1	R-10	R-20	mAP	R-1	R-10	R-20	mAP	R-1	R-10	R-20	mAP
HOG^[21]	2005	2.76	18.30	31.90	4.24	3.82	22.80	37.60	2.16	3.22	24.70	44.50	7.25	4.75	29.20	49.40	3.51
LOMO^[22]	2015	3.64	23.20	37.30	4.53	4.70	28.20	43.10	2.28	5.75	34.40	54.90	10.20	7.36	40.40	60.30	5.64
Zero-Padding^[6]	2017	14.80	54.10	71.30	15.90	19.10	61.40	78.40	10.90	20.60	68.40	85.80	26.90	24.40	75.90	91.30	18.60
D-HSME^[14]	2019	20.68	62.74	77.95	23.12	-	-	-	-	-	-	-	-	-	-	-	-
cmGAN^[9]	2018	27.00	67.50	80.60	27.80	31.50	72.70	85.00	22.30	31.60	77.20	89.20	42.20	37.00	80.90	92.10	32.80
eBDTR^[13]	2020	27.82	67.34	81.34	28.42	-	-	-	-	-	-	-	-	-	-	-	-
D²RL^[10]	2019	28.90	70.60	82.40	29.20	-	-	-	-	-	-	-	-	-	-	-	-
JSIA-ReID^[23]	2020	38.10	80.70	89.90	36.90	45.10	85.70	93.80	29.50	43.80	86.20	94.20	52.90	52.70	91.10	96.40	42.70
AlignGAN^[11]	2019	42.40	85.00	93.70	40.70	51.50	89.40	95.70	33.90	45.90	87.60	94.40	54.30	57.10	92.70	97.40	45.30
X modality^[16]	2020	49.92	89.79	95.96	50.73	-	-	-	-	-	-	-	-	-	-	-	-
MACE^[26]	2020	51.64	87.25	94.44	50.11	-	-	-	-	57.35	93.02	97.47	64.79	-	-	-	-
DDAG^[17]	2020	54.75	90.39	95.81	53.02	-	-	-	-	61.02	94.06	98.41	67.98	-	-	-	-
SIM^[15]	2020	56.93	-	-	60.88	-	-	-	-	-	-	-	-	-	-	-	-
NFS^[18]	2021	56.91	91.34	96.52	55.45	63.51	94.42	97.81	48.56	62.79	96.53	99.07	69.79	70.03	97.70	99.51	61.45
CICL^[12]	2021	57.20	94.30	98.40	59.30	60.70	95.20	98.60	52.60	66.60	98.80	99.70	74.70	73.80	99.40	99.90	68.30
cm-SSFT^[27]	2020	61.60	89.20	93.90	63.20	63.40	91.20	95.70	62.00	70.50	94.90	97.70	72.60	73.00	96.30	99.10	72.40
CANet^[28]	2021	69.88	95.71	98.46	66.89	-	-	-	-	76.26	97.88	99.49	80.37	-	-	-	-
本文方法	2022	68.53	95.80	97.87	66.27	66.81	95.63	98.17	62.80	74.65	97.02	99.21	79.33	78.91	97.53	99.14	75.27

方法	年份	All-search (%)								Indoor-search (%)
		Single-shot				Multi-shot				Single-shot				Multi-shot
		R-1	R-10	R-20	mAP	R-1	R-10	R-20	mAP	R-1	R-10	R-20	mAP	R-1	R-10	R-20	mAP
HOG^[21]	2005	2.76	18.30	31.90	4.24	3.82	22.80	37.60	2.16	3.22	24.70	44.50	7.25	4.75	29.20	49.40	3.51
LOMO^[22]	2015	3.64	23.20	37.30	4.53	4.70	28.20	43.10	2.28	5.75	34.40	54.90	10.20	7.36	40.40	60.30	5.64
Zero-Padding^[6]	2017	14.80	54.10	71.30	15.90	19.10	61.40	78.40	10.90	20.60	68.40	85.80	26.90	24.40	75.90	91.30	18.60
D-HSME^[14]	2019	20.68	62.74	77.95	23.12	-	-	-	-	-	-	-	-	-	-	-	-
cmGAN^[9]	2018	27.00	67.50	80.60	27.80	31.50	72.70	85.00	22.30	31.60	77.20	89.20	42.20	37.00	80.90	92.10	32.80
eBDTR^[13]	2020	27.82	67.34	81.34	28.42	-	-	-	-	-	-	-	-	-	-	-	-
D²RL^[10]	2019	28.90	70.60	82.40	29.20	-	-	-	-	-	-	-	-	-	-	-	-
JSIA-ReID^[23]	2020	38.10	80.70	89.90	36.90	45.10	85.70	93.80	29.50	43.80	86.20	94.20	52.90	52.70	91.10	96.40	42.70
AlignGAN^[11]	2019	42.40	85.00	93.70	40.70	51.50	89.40	95.70	33.90	45.90	87.60	94.40	54.30	57.10	92.70	97.40	45.30
X modality^[16]	2020	49.92	89.79	95.96	50.73	-	-	-	-	-	-	-	-	-	-	-	-
MACE^[26]	2020	51.64	87.25	94.44	50.11	-	-	-	-	57.35	93.02	97.47	64.79	-	-	-	-
DDAG^[17]	2020	54.75	90.39	95.81	53.02	-	-	-	-	61.02	94.06	98.41	67.98	-	-	-	-
SIM^[15]	2020	56.93	-	-	60.88	-	-	-	-	-	-	-	-	-	-	-	-
NFS^[18]	2021	56.91	91.34	96.52	55.45	63.51	94.42	97.81	48.56	62.79	96.53	99.07	69.79	70.03	97.70	99.51	61.45
CICL^[12]	2021	57.20	94.30	98.40	59.30	60.70	95.20	98.60	52.60	66.60	98.80	99.70	74.70	73.80	99.40	99.90	68.30
cm-SSFT^[27]	2020	61.60	89.20	93.90	63.20	63.40	91.20	95.70	62.00	70.50	94.90	97.70	72.60	73.00	96.30	99.10	72.40
CANet^[28]	2021	69.88	95.71	98.46	66.89	-	-	-	-	76.26	97.88	99.49	80.37	-	-	-	-
本文方法	2022	68.53	95.80	97.87	66.27	66.81	95.63	98.17	62.80	74.65	97.02	99.21	79.33	78.91	97.53	99.14	75.27

方法	Single-shot and All-search
方法	Rank-1	mAP
Baseline	58.17	56.69
Raw-SpaceAttention	65.67	62.50
Raw-ChannelAttention	63.43	62.41
Raw-DuanlAttention	69.53	67.52
Residual-SpaceAttention	68.34	66.57
Residual-ChannelAttention	67.80	66.23
Residual-DuanlAttention	68.53	66.27

方法	Single-shot and All-search
方法	Rank-1	mAP
Baseline	58.17	56.69
Raw-SpaceAttention	65.67	62.50
Raw-ChannelAttention	63.43	62.41
Raw-DuanlAttention	69.53	67.52
Residual-SpaceAttention	68.34	66.57
Residual-ChannelAttention	67.80	66.23
Residual-DuanlAttention	68.53	66.27

方法	Single-shot and All-search
方法	Rank-1	mAP
参数共享的单流网络	54.33	51.19
参数完全独立的双流网络	19.50	44.70
本文的网络结构	58.17	56.69