Orthogonal fusion image descriptor based on global attention

doi:10.11996/JG.j.2095-302X.2024030472

Abstract

Abstract:

Image descriptors are important research objects in computer vision tasks and are widely applied to the fields of image classification, segmentation, recognition, and retrieval. The depth image descriptor lacks the correlation between the high-dimensional feature space and channel information in the local feature extraction branch, resulting in insufficient information for local feature expression. Therefore, an image descriptor combining local and global features was proposed. The multi-scale feature map was extracted through dilated convolution in the local feature extraction branch. After the output features were spliced, the relevant channel-space information was captured through a global attention mechanism with a multilayer perceptron. Then the final local features were output after processing. The high-dimensional global branches generated global feature vectors through global pooling and full convolution. The orthogonal values of local features were extracted on the global feature vector, and were then concatenated with the global features to form the final descriptor. At the same time, the robustness of the model in large-scale datasets were enhanced by employing the angular domain loss function containing the sub-class center. The experimental results on the publicly available datasets Roxford5k and Rparis6k demonstrated that in medium and hard modes, the average retrieval accuracy of this descriptor reached 81.87% and 59.74%, and 91.61% and 79.12%, respectively. This represented an improvement of 1.70% and 1.56%, and 2.00% and 1.83% compared to that of deep orthogonal fusion descriptors. It exhibited superior retrieval accuracy over other image descriptors.

Key words: image descriptor, dilated convolution, global attention, feature fusion, sub-center arcface

CLC Number:

TP391

AI Liefu, TAO Yong, JIANG Changyu. Orthogonal fusion image descriptor based on global attention[J]. Journal of Graphics, 2024, 45(3): 472-481.

Figures/Tables 11

Fig. 1 Descriptor formation structure diagram

Fig. 2 Branch structure diagram for local feature extraction

Fig. 3 GAM structure chart

Fig. 4 Structure diagram of GAM channel attention submodule

Fig. 5 Structure diagram of GAM spatial attention submodule

Fig. 6 Result graph of different number of subclasses of loss function

Fig. 7 Result graph of global pooling with different p-values

Table 1 Experimental results of eliminating partial modules/%

消融实验	Roxf- medium	Roxf- hard	Rpar- medium	Rpar- hard
空洞卷积(×)	81.81	59.35	89.87	79.08
全局注意力(×)	81.44	59.18	91.13	78.87
自注意力(×)	80.84	58.67	90.77	78.34
GA-DOLG	81.87	59.74	91.61	79.12

Table 2 Experimental results of different descriptor algorithms/%

描述符	Roxf- medium	Roxf- hard	Rpar- medium	Rpar- hard
DELF	76.00	52.40	80.20	58.60
ASMK	79.10	52.70	91.00	81.00
DELG	79.08	58.40	88.78	76.20
How-ASMK	79.40	56.90	81.60	62.40
Hot-Refresh	67.34	53.28	81.63	68.96
DOLG	80.50	58.82	89.81	77.70
GA-DOLG	81.87	59.74	91.61	79.12

Table 3 mAP@10 Experimental results/%

描述符	Roxf- medium	Roxf- hard	Rpar- medium	Rpar- hard
DOLG	92.57	71.14	98.43	93.71
GA-DOLG	93.76	72.27	99.41	94.17

Fig. 8 Visualization results of descriptors ((a) Image retrieval 1; (b) Image retrieval 2)

References 38

[1]	LOWE D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91-110.
[2]	BAY H, TUYTELAARS T, VAN GOOL L. SURF: speeded up robust features[M]//Computer Vision - ECCV 2006. Heidelberg: Springer, 2006: 404-417.
[3]	SIVIC J, ZISSERMAN A. Video Google: a text retrieval approach to object matching in videos[C]// The 9th IEEE International Conference on Computer Vision. New York: IEEE Press, 2008: 1470-1477.
[4]	PERRONNIN F, DANCE C. Fisher kernels on visual vocabularies for image categorization[C]// 2007 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2007: 1-8.
[5]	JÉGOU H, DOUZE M, SCHMID C, et al. Aggregating local descriptors into a compact image representation[C]// 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2010: 3304-3311.
[6]	魏本昌, 郑丽, 管涛. 残差增强的图像描述符[J]. 计算机辅助设计与图形学学报, 2019, 31(6): 1039-1045.
	WEI B C, ZHENG L, GUAN T. Residual enhanced image descriptor[J]. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(6): 1039-1045 (in Chinese).
[7]	JÉGOU H, DOUZE M, SCHMID C. Product quantization for nearest neighbor search[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(1): 117-128. DOI PMID
[8]	WANG J D, ZHANG T, SONG J K, et al. A survey on learning to hash[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 769-790. DOI PMID
[9]	BEIS J S, LOWE D G. Shape indexing using approximate nearest-neighbour search in high-dimensional spaces[C]// IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2002: 1000-1006.
[10]	NISTER D, STEWENIUS H. Scalable recognition with a vocabulary tree[C]// 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2006: 2161-2168.
[11]	吴泽斌, 于俊清, 何云峰, 等. 一种用于图像检索的多层语义二值描述符[J]. 计算机学报, 2020, 43(9): 1641-1655.
	WU Z B, YU J Q, HE Y F, et al. Multi-level semantic binary descriptor for image retrieval[J]. Chinese Journal of Computers, 2020, 43(9): 1641-1655 (in Chinese).
[12]	SIMÉONI O, AVRITHIS Y, CHUM O. Local features and visual words emerge in activations[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 11643-11652.
[13]	FISCHLER M A, BOLLES R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[M]// Readings in Computer Vision. Amsterdam: Elsevier, 1987: 726-740.
[14]	TOLIAS G, AVRITHIS Y, JÉGOU H. Image search with selective match kernels: aggregation across single and multiple images[J]. International Journal of Computer Vision, 2016, 116(3): 247-261.
[15]	NOH H, ARAUJO A, SIM J, et al. Large-scale image retrieval with attentive deep local features[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 3476-3485.
[16]	CAO B Y, ARAUJO A, SIM J. Unifying deep local and global features for image search[C]// European Conference on Computer Vision. Cham: Springer, 2020: 726-743.
[17]	ÖZTÜRK Ş, ÇELIK E, ÇUKUR T. Content-based medical image retrieval with opponent class adaptive margin loss[J]. Information Sciences, 2023, 637: 118938.
[18]	ARANDJELOVIĆ R, GRONAT P, TORII A, et al. NetVLAD: CNN architecture for weakly supervised place recognition[C]// IEEE Transactions on Pattern Analysis and Machine Intelligence. New York: IEEE Press, 2018: 1437-1451.
[19]	HAUSLER S, GARG S, XU M, et al. Patch-NetVLAD: multi-scale fusion of locally-global descriptors for place recognition[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 14136-14147.
[20]	TOLIAS G, JENICEK T, CHUM O. Learning and aggregating deep local descriptors for instance-level recognition[C]// European Conference on Computer Vision. Cham: Springer, 2020: 460-477.
[21]	RADENOVIC F, TOLIAS G, CHUM O. Fine-tuning CNN image retrieval with No human annotation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(7): 1655-1668. DOI PMID
[22]	ZHU Y Y, CAO G, YANG Z Y, et al. Learning relation-based features for fine-grained image retrieval[J]. Pattern Recognition, 2023, 140: 109543.
[23]	YANG M, HE D L, FAN M, et al. DOLG: single-stage image retrieval with deep orthogonal fusion of local and global features[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 11752-11761.
[24]	LIU Y C, SHAO Z R, HOFFMANN N. Global attention mechanism: retain information to enhance channel-spatial interactions[EB/OL]. [2023-01-23]. http://arxiv.org/abs/2112.05561.pdf.
[25]	WANG P, LI X, YARAS C, et al. Understanding deep representation learning via layerwise feature compression and discrimination[EB/OL]. [2023-01-23]. http://arxiv.org/abs/2311.02960.pdf.
[26]	DENG J K, GUO J, XUE N N, et al. ArcFace: additive angular margin loss for deep face recognition[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 4685-4694.
[27]	DENG J K, GUO J, LIU T L, et al. Sub-center ArcFace: boosting face recognition by large-scale noisy web faces[C]// European Conference on Computer Vision. Cham: Springer, 2020: 741-757.
[28]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7132-7141.
[29]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]// European Conference on Computer Vision. Cham: Springer, 2018: 3-19.
[30]	PARK J, WOO S, LEE J Y, et al. BAM: bottleneck attention module[EB/OL]. [2023-01-23]. http://arxiv.org/abs/1807.06514.pdf.
[31]	MISRA D, NALAMADA T, ARASANIPALAI A U, et al. Rotate to attend: convolutional triplet attention module[C]// 2021 IEEE Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2021: 3138-3147.
[32]	CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL]. [2023-01-23]. http://arxiv.org/abs/1706.05587.pdf.
[33]	NOH H, ARAUJO A, SIM J, et al. Large-scale image retrieval with attentive deep local features[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 3476-3485.
[34]	QIN Q, HU W P, LIU B. Feature projection for improved text classification[C]// The 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2020: 8161-8171.
[35]	HE K M, ZHANG X Y, REN S Q, et al. Identity mappings in deep residual networks[M]. Computer Vision - ECCV 2016. Cham: Springer International Publishing, 2016: 630-645.
[36]	WEYAND T, ARAUJO A, CAO B Y, et al. Google landmarks dataset v2-A large-scale benchmark for instance-level recognition and retrieval[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 2572-2581.
[37]	RADENOVIC F, ISCEN A, TOLIAS G, et al. Revisiting Oxford and Paris: large-scale image retrieval benchmarking[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 5706-5715.
[38]	ZHANG B J, GE Y X, SHEN Y T, et al. Hot-refresh model upgrades with regression-alleviating compatible training in image retrieval[EB/OL]. [2023-01-23]. http://arxiv.org/abs/2201.09724.pdf.