Graph element detection matching based on Republic of China banknotes

doi:10.11996/JG.j.2095-302X.2023030492

Abstract

Abstract:

In view of the fact that there are numerous types of Republic of China banknotes, which often have slight visual differences between different banknote, combined with the issues of mold, burrs or breakage after circulation, the recognition and classification ability of traditional fine-grained image retrieval methods for Republican banknotes is inadequate. To address these issues, this paper proposed a fine-grained retrieval model of Republican banknotes based on multiscale feature fusion. To reduce the time of manual data labeling, YOLOv4 was employed for graph element detection on banknote images, with the main view of banknotes being adopted as the input feature map. EfficientNet-B0 was utilized as the backbone network for retrieval, thereby reducing the burden of redundant information in the network and enhancing network accuracy. In the model, the feature vectors of layers 2, 4, 10, and 15 of the PANet fusion network were utilized to generate a global feature vector library, improving the banknote matching retrieval capability. Furthermore, the feature vectors were clustered using adaptive K-means to simplify the matching time and computation. The experimental results demonstrated that the proposed model achieved an accuracy of 89.6%, improving the retrieval accuracy by 10 percentage points compared to using the original image of banknotes as the input image. The improved model exhibited better classification performance, less inference time cost, and fine classification of banknotes. These results could meet the practical requirements of industry.

Key words: banknotes of the Republic of China, deep learning, object detection, image retrieval, fine-grained image classification

CLC Number:

WANG Jia-jing, WANG Chen, ZHU Yuan-yuan, WANG Xiao-mei. Graph element detection matching based on Republic of China banknotes[J]. Journal of Graphics, 2023, 44(3): 492-501.

Figures/Tables 19

Fig. 1 Example of elements in the banknote

Fig. 2 Classification of main view

Fig. 3 Flow chart of banknote retrieval

Fig. 4 YOLOv4 detection framework diagram

Fig. 5 Feature extraction and feature fusion module structure

Fig. 6 Image pre-processing ((a) Unprocessed images of original banknotes; (b) Image after Gaussian filtering; (c) Results after histogram equalization; (d) Results after Hough transformation and tilt correction)

Table 1 Detection of network experimental parameter settings

参数	数值
输入数据批大小	64
输入原图尺寸	416×416×3
动量系数	0.949
权重衰减正则系数	0.000 5
学习率	0.001
最大迭代次数	8 000
学习率变动步长	6 400，7 200
学习率变动因子	0.1
类别数	5
滤波器数量	27

Fig. 7 Example graph of model detection results

Fig. 8 The loss curve during model training

Table 2 Comparison of detection and recognition performances of each model

检测模型	AP (%)					mAP (%)	Time (s)
检测模型	角花	花符	印章	签名	主景图	mAP (%)	Time (s)
YOLOv3	88.32	87.27	84.89	64.24	81.22	81.19	0.294
SSD	-	-	-	-	-	70.71	1.369
Faster R-CNN	83.34	80.76	89.74	62.68	80.31	79.35	3.626
YOLOv4	96.49	93.16	91.53	82.76	95.26	91.84	0.373

Fig. 9 Recall curves for model detection of main view categories

Fig. 10 Model detection of PR curves for main view categories

Fig. 11 Main view detection results (part)

Table 3 Feature extraction network parameter settings

参数	数值
输入数据批大小	32
输入组合尺寸	224×224×3
动量系数	0.9
权重衰减正则系数	0.000 1
学习率	0.001
最大迭代次数	90
学习率变动因子	0.1
类别数	129

Table 4 Comparison results of classification networks (%)

模型名称	Top-1 ACC	Top-5 ACC
AlexNet	28.492	83.631
VGG-16	24.302	82.961
ResNet-50	40.447	86.648
EfficientNet-B0	81.229	97.039
MobileNet-V2	80.637	96.855
GoogLeNet	78.665	93.748
改进后的EfficientNet-B0	86.793	98.198

Fig. 12 Clustering results of feature vector library

Table 5 Comparative experimental results of different EfficientNet models

模型名称	mAP (%)	参数量(MB)
EfficientNet-B0 (原图)	79.53	32.8
EfficientNet-B3 (原图)	81.58	86.6
EfficientNet-B7 (原图)	82.09	149.6
EfficientNet-B0 (主景图+匹配)	89.60	32.8
EfficientNet-B3 (主景图+匹配)	90.48	86.6
EfficientNet-B7 (主景图+匹配)	91.89	149.6

Fig. 13 Image to be matched

Fig. 14 Comparison experiment results of image matching of banknotes in the Republic of China ((a) Feature extraction matching results of main view; (b) Feature extraction and matching results of original banknote image)

References 20

[1]	PARASHIVAMURTHY R, NAVEENA C, SHARATH KUMAR Y H. SIFT and HOG features for the retrieval of ancient Kannada epigraphs[J]. IET Image Processing, 2020, 14(17): 4657-4662. DOI URL
[2]	LIU H, ZHAO Q J, ZHANG C, et al. Boosting VLAD with weighted fusion of local descriptors for image retrieval[J]. Multimedia Tools and Applications, 2019, 78(9): 11835-11855. DOI
[3]	KISHORE D, RAO C. A multi-class SVM based content based image retrieval system using hybrid optimization techniques[J]. Traitement Du Signal, 2020, 37(2): 217-226. DOI URL
[4]	GHRABAT M J J, MA G Z, MAOLOOD I Y, et al. An effective image retrieval based on optimized genetic algorithm utilized a novel SVM-based convolutional neural network classifier[J]. Human-Centric Computing and Information Sciences, 2019, 9(1): 1-29. DOI
[5]	GE Z X, CAO G, LI X S, et al. Hyperspectral image classification method based on 2D-3D CNN and multibranch feature fusion[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 5776-5788. DOI URL
[6]	王志伟, 普园媛, 王鑫, 等. 基于多特征融合的多尺度服装图像精准化检索[J]. 计算机学报, 2020, 43(4): 740-754.
	WANG Z W, PU Y Y, WANG X, et al. Accurate retrieval of multi-scale clothing images based on multi-feature fusion[J]. Chinese Journal of Computers, 2020, 43(4): 740-754. (in Chinese)
[7]	周书仁, 谢盈, 蔡碧野. 融合多尺度特征的深度哈希图像检索方法[J]. 计算机科学与探索, 2018, 12(12): 1974-1986. DOI
	ZHOU S R, XIE Y, CAI B Y. Deep hashing method for image retrieval based on multi-scale features[J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(12): 1974-1986. (in Chinese)
[8]	HE K M, GKIOXARI G, DOLLAR P, et al. Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 386-397. DOI PMID
[9]	DUBEY A, GUPTA O, GUO P, et al. Pairwise confusion for fine-grained visual classification[EB/OL]. [2022-05-08]. https://arxiv.org/abs/1705.08016.
[10]	顾军华, 王锋, 戚永军, 等. 基于多尺度卷积特征融合的肺结节图像检索方法[J]. 计算机应用, 2020, 40(2): 561-565. DOI
	GU J H, WANG F, QI Y J, et al. Retrieval method of pulmonary nodule images based on multi-scale convolution feature fusion[J]. Journal of Computer Applications, 2020, 40(2): 561-565. (in Chinese) DOI
[11]	朱明, 汪桐生, 王年, 等. 基于多尺度自注意卷积的足迹压力图像检索算法[J]. 模式识别与人工智能, 2020, 33(12): 1097-1103. DOI
	ZHU M, WANG T S, WANG N, et al. Footprint pressure image retrieval algorithm based on multi-scale self-attention convolution[J]. Pattern Recognition and Artificial Intelligence, 2020, 33(12): 1097-1103. (in Chinese) DOI
[12]	ZHANG F, LI M, ZHAI G S, et al. Multi-branch and multi-scale attention learning for fine-grained visual categorization[M]//MultiMedia Modeling. Cham: Springer International Publishing, 2021: 136-147.
[13]	LI A X, HUANG W R, LAN X, et al. Boosting few-shot learning with adaptive margin loss[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 12573-12581.
[14]	LI M, ZHOU G, CAI W, et al. Multi-scale sparse network with cross-attention mechanism for image-based butterflies fine-grained classification[EB/OL]. (2022-03-01) [2022-05-14]. https://www.sciencedirect.com/science/article/pii/S1568494622000060?via%3Dihub.
[15]	LYU C Z, HU G Q, WANG D. Attention to fine-grained information: hierarchical multi-scale network for retinal vessel segmentation[J]. The Visual Computer, 2022, 38(1): 345-355. DOI
[16]	SINHA A, DOLZ J. Multi-scale self-guided attention for medical image segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 25(1): 121-130. DOI URL
[17]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848. DOI URL
[18]	WEI X S. Mask-CNN: localizing parts and selecting descriptors for fine-grained bird species categorization[J]. Pattern Recognition, 2018, 76: 704-714. DOI URL
[19]	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. Scaled-YOLOv4: scaling cross stage partial network[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 13024-13033.
[20]	WU T T, GU X Y, SHAO J B, et al. Colour image segmentation based on a convex K-means approach[J]. IET Image Processing, 2021, 15(8): 1596-1606. DOI URL