基于民国纸币的图元素匹配检索

doi:10.11996/JG.j.2095-302X.2023030492

图学学报 ›› 2023, Vol. 44 ›› Issue (3): 492-501.DOI: 10.11996/JG.j.2095-302X.2023030492

• 图像处理与计算机视觉 • 上一篇下一篇

基于民国纸币的图元素匹配检索

王佳婧(), 王晨, 朱媛媛, 王笑梅()

上海师范大学信息与机电工程学院，上海 200234

收稿日期:2022-10-31 接受日期:2022-12-19 出版日期:2023-06-30 发布日期:2023-06-30
通讯作者: 王笑梅(1970-)，女，副教授，硕士。主要研究方向为图像处理与计算机网络等。E-mail：xiaomei@shnu.edu.cn
作者简介:
王佳婧(1998-)，女，硕士研究生。主要研究方向为计算机视觉。E-mail：13262267327@163.com
基金资助:
馆藏文物预防性保护风险防控关键技术研究示范项目(2020YFC152250);中央宣传部文化名家暨四个一批人才工程“民国纸币研究项目”

Graph element detection matching based on Republic of China banknotes

WANG Jia-jing(), WANG Chen, ZHU Yuan-yuan, WANG Xiao-mei()

School of Information and Electromechanical Engineering, Shanghai Normal University, Shanghai 200234, China

Received:2022-10-31 Accepted:2022-12-19 Online:2023-06-30 Published:2023-06-30
Contact: WANG Xiao-mei (1970-), associate professor, master. Her main research interests cover image processing and computer network, etc. E-mail：xiaomei@shnu.edu.cn
About author:
WANG Jia-jing (1998-), master student. Her main research interest covers computer vision. E-mail：13262267327@163.com
Supported by:
Research Demonstration on Key Technology of Preventive Protection and Risk Prevention and Control of Cultural Relics in Collections(2020YFC152250);Cultural Masters of the Central Propaganda Department and four batches of talent projects “Republic of China Paper Currency Research Project”

摘要/Abstract

摘要：

民国纸币种类数量众多，不同纸币类别间的视觉差异小，部分纸币经过流通后发霉、毛边以及破损。针对传统的细粒度图像检索方法对民国纸币识别分类能力差的问题，提出了一种基于多尺度特征融合的民国纸币细粒度检索模型。在使用YOLOv4对纸币图像做图元素检测，减少手动标记数据时间的基础上，利用纸币主景图作为输入特征图，使用EfficientNet-B0作为主干网络进行检索，减少了冗余信息对网络的负担，提升了网络的精度。在模型中，使用PANet融合网络的第2，4，10和15层的特征向量，生成全局特征向量库，提升了纸币匹配检索能力，并使用自适应K均值对特征向量进行聚类，简化了匹配的时间与计算量。实验结果表明，该模型准确率达到了89.6%，相比于使用纸币原图作为输入图像提升了10个百分点，提高了检索精度。改进后的模型分类效果更好，推理时间成本更少，实现了纸币的精细化分类。满足工业实际要求。

关键词: 民国纸币, 深度学习, 目标检测, 图像检索, 细粒度图像分类

Abstract:

In view of the fact that there are numerous types of Republic of China banknotes, which often have slight visual differences between different banknote, combined with the issues of mold, burrs or breakage after circulation, the recognition and classification ability of traditional fine-grained image retrieval methods for Republican banknotes is inadequate. To address these issues, this paper proposed a fine-grained retrieval model of Republican banknotes based on multiscale feature fusion. To reduce the time of manual data labeling, YOLOv4 was employed for graph element detection on banknote images, with the main view of banknotes being adopted as the input feature map. EfficientNet-B0 was utilized as the backbone network for retrieval, thereby reducing the burden of redundant information in the network and enhancing network accuracy. In the model, the feature vectors of layers 2, 4, 10, and 15 of the PANet fusion network were utilized to generate a global feature vector library, improving the banknote matching retrieval capability. Furthermore, the feature vectors were clustered using adaptive K-means to simplify the matching time and computation. The experimental results demonstrated that the proposed model achieved an accuracy of 89.6%, improving the retrieval accuracy by 10 percentage points compared to using the original image of banknotes as the input image. The improved model exhibited better classification performance, less inference time cost, and fine classification of banknotes. These results could meet the practical requirements of industry.

Key words: banknotes of the Republic of China, deep learning, object detection, image retrieval, fine-grained image classification

中图分类号:

王佳婧, 王晨, 朱媛媛, 王笑梅. 基于民国纸币的图元素匹配检索[J]. 图学学报, 2023, 44(3): 492-501.

WANG Jia-jing, WANG Chen, ZHU Yuan-yuan, WANG Xiao-mei. Graph element detection matching based on Republic of China banknotes[J]. Journal of Graphics, 2023, 44(3): 492-501.

图/表 19

图1 民国纸币图元素示例

Fig. 1 Example of elements in the banknote

图2 民国纸币主景图的分类统计

Fig. 2 Classification of main view

图3 民国纸币检索流程图

Fig. 3 Flow chart of banknote retrieval

图4 YOLOv4检测框架图

Fig. 4 YOLOv4 detection framework diagram

图5 特征提取和特征融合模块结构

Fig. 5 Feature extraction and feature fusion module structure

图6 图像预处理((a)为未经处理的原始纸币图像；(b)为高斯滤波处理后的图像；(c)为直方图均衡化的结果图；(d)经过霍夫变换和倾斜校正的结果图)

Fig. 6 Image pre-processing ((a) Unprocessed images of original banknotes; (b) Image after Gaussian filtering; (c) Results after histogram equalization; (d) Results after Hough transformation and tilt correction)

表1 检测网络实验参数设置

Table 1 Detection of network experimental parameter settings

参数	数值
输入数据批大小	64
输入原图尺寸	416×416×3
动量系数	0.949
权重衰减正则系数	0.000 5
学习率	0.001
最大迭代次数	8 000
学习率变动步长	6 400，7 200
学习率变动因子	0.1
类别数	5
滤波器数量	27

图7 模型检测结果示例图

Fig. 7 Example graph of model detection results

图8 模型训练过程中的损失曲线

Fig. 8 The loss curve during model training

表2 各模型检测识别性能比较

Table 2 Comparison of detection and recognition performances of each model

检测模型	AP (%)					mAP (%)	Time (s)
检测模型	角花	花符	印章	签名	主景图	mAP (%)	Time (s)
YOLOv3	88.32	87.27	84.89	64.24	81.22	81.19	0.294
SSD	-	-	-	-	-	70.71	1.369
Faster R-CNN	83.34	80.76	89.74	62.68	80.31	79.35	3.626
YOLOv4	96.49	93.16	91.53	82.76	95.26	91.84	0.373

图9 模型检测主景图类别的召回曲线

Fig. 9 Recall curves for model detection of main view categories

图10 模型检测主景图类别的PR曲线

Fig. 10 Model detection of PR curves for main view categories

图11 主景图检测结果(部分)

Fig. 11 Main view detection results (part)

表3 特征提取网络参数设置

Table 3 Feature extraction network parameter settings

参数	数值
输入数据批大小	32
输入组合尺寸	224×224×3
动量系数	0.9
权重衰减正则系数	0.000 1
学习率	0.001
最大迭代次数	90
学习率变动因子	0.1
类别数	129

表4 分类网络对比结果(%)

Table 4 Comparison results of classification networks (%)

模型名称	Top-1 ACC	Top-5 ACC
AlexNet	28.492	83.631
VGG-16	24.302	82.961
ResNet-50	40.447	86.648
EfficientNet-B0	81.229	97.039
MobileNet-V2	80.637	96.855
GoogLeNet	78.665	93.748
改进后的EfficientNet-B0	86.793	98.198

图12 特征向量库聚类结果

Fig. 12 Clustering results of feature vector library

表5 不同EfficientNet模型的对比实验结果

Table 5 Comparative experimental results of different EfficientNet models

模型名称	mAP (%)	参数量(MB)
EfficientNet-B0 (原图)	79.53	32.8
EfficientNet-B3 (原图)	81.58	86.6
EfficientNet-B7 (原图)	82.09	149.6
EfficientNet-B0 (主景图+匹配)	89.60	32.8
EfficientNet-B3 (主景图+匹配)	90.48	86.6
EfficientNet-B7 (主景图+匹配)	91.89	149.6

图13 待匹配民国纸币图像

Fig. 13 Image to be matched

图14 民国纸币图像匹配对照实验结果((a)主景图特征提取匹配结果；(b)纸币原图特征提取匹配结果)

Fig. 14 Comparison experiment results of image matching of banknotes in the Republic of China ((a) Feature extraction matching results of main view; (b) Feature extraction and matching results of original banknote image)

参考文献 20

[1]	PARASHIVAMURTHY R, NAVEENA C, SHARATH KUMAR Y H. SIFT and HOG features for the retrieval of ancient Kannada epigraphs[J]. IET Image Processing, 2020, 14(17): 4657-4662. DOI URL
[2]	LIU H, ZHAO Q J, ZHANG C, et al. Boosting VLAD with weighted fusion of local descriptors for image retrieval[J]. Multimedia Tools and Applications, 2019, 78(9): 11835-11855. DOI
[3]	KISHORE D, RAO C. A multi-class SVM based content based image retrieval system using hybrid optimization techniques[J]. Traitement Du Signal, 2020, 37(2): 217-226. DOI URL
[4]	GHRABAT M J J, MA G Z, MAOLOOD I Y, et al. An effective image retrieval based on optimized genetic algorithm utilized a novel SVM-based convolutional neural network classifier[J]. Human-Centric Computing and Information Sciences, 2019, 9(1): 1-29. DOI
[5]	GE Z X, CAO G, LI X S, et al. Hyperspectral image classification method based on 2D-3D CNN and multibranch feature fusion[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 5776-5788. DOI URL
[6]	王志伟, 普园媛, 王鑫, 等. 基于多特征融合的多尺度服装图像精准化检索[J]. 计算机学报, 2020, 43(4): 740-754.
	WANG Z W, PU Y Y, WANG X, et al. Accurate retrieval of multi-scale clothing images based on multi-feature fusion[J]. Chinese Journal of Computers, 2020, 43(4): 740-754. (in Chinese)
[7]	周书仁, 谢盈, 蔡碧野. 融合多尺度特征的深度哈希图像检索方法[J]. 计算机科学与探索, 2018, 12(12): 1974-1986. DOI
	ZHOU S R, XIE Y, CAI B Y. Deep hashing method for image retrieval based on multi-scale features[J]. Journal of Frontiers of Computer Science and Technology, 2018, 12(12): 1974-1986. (in Chinese)
[8]	HE K M, GKIOXARI G, DOLLAR P, et al. Mask R-CNN[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 386-397. DOI PMID
[9]	DUBEY A, GUPTA O, GUO P, et al. Pairwise confusion for fine-grained visual classification[EB/OL]. [2022-05-08]. https://arxiv.org/abs/1705.08016.
[10]	顾军华, 王锋, 戚永军, 等. 基于多尺度卷积特征融合的肺结节图像检索方法[J]. 计算机应用, 2020, 40(2): 561-565. DOI
	GU J H, WANG F, QI Y J, et al. Retrieval method of pulmonary nodule images based on multi-scale convolution feature fusion[J]. Journal of Computer Applications, 2020, 40(2): 561-565. (in Chinese) DOI
[11]	朱明, 汪桐生, 王年, 等. 基于多尺度自注意卷积的足迹压力图像检索算法[J]. 模式识别与人工智能, 2020, 33(12): 1097-1103. DOI
	ZHU M, WANG T S, WANG N, et al. Footprint pressure image retrieval algorithm based on multi-scale self-attention convolution[J]. Pattern Recognition and Artificial Intelligence, 2020, 33(12): 1097-1103. (in Chinese) DOI
[12]	ZHANG F, LI M, ZHAI G S, et al. Multi-branch and multi-scale attention learning for fine-grained visual categorization[M]//MultiMedia Modeling. Cham: Springer International Publishing, 2021: 136-147.
[13]	LI A X, HUANG W R, LAN X, et al. Boosting few-shot learning with adaptive margin loss[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 12573-12581.
[14]	LI M, ZHOU G, CAI W, et al. Multi-scale sparse network with cross-attention mechanism for image-based butterflies fine-grained classification[EB/OL]. (2022-03-01) [2022-05-14]. https://www.sciencedirect.com/science/article/pii/S1568494622000060?via%3Dihub.
[15]	LYU C Z, HU G Q, WANG D. Attention to fine-grained information: hierarchical multi-scale network for retinal vessel segmentation[J]. The Visual Computer, 2022, 38(1): 345-355. DOI
[16]	SINHA A, DOLZ J. Multi-scale self-guided attention for medical image segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 25(1): 121-130. DOI URL
[17]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848. DOI URL
[18]	WEI X S. Mask-CNN: localizing parts and selecting descriptors for fine-grained bird species categorization[J]. Pattern Recognition, 2018, 76: 704-714. DOI URL
[19]	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. Scaled-YOLOv4: scaling cross stage partial network[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 13024-13033.
[20]	WU T T, GU X Y, SHAO J B, et al. Colour image segmentation based on a convex K-means approach[J]. IET Image Processing, 2021, 15(8): 1596-1606. DOI URL

基于民国纸币的图元素匹配检索

Graph element detection matching based on Republic of China banknotes

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 19

参考文献 20

相关文章 15

编辑推荐

Metrics

本文评价

[1]	毕春艳, 刘越. 基于深度学习的视频人体动作识别综述[J]. 图学学报, 2023, 44(4): 625-639.
[2]	李利霞, 王鑫, 王军, 张又元 . 基于特征融合与注意力机制的无人机图像小目标检测算法 [J]. 图学学报, 2023, 44(4): 658-666.
[3]	曹义亲 , 周一纬 , 徐露 . 基于 E-YOLOX 的实时金属表面缺陷检测算法 [J]. 图学学报, 2023, 44(4): 677-690.
[4]	邵俊棋, 钱文华, 徐启豪. 基于条件残差生成对抗网络的风景图生成 [J]. 图学学报, 2023, 44(4): 710-717.
[5]	邓渭铭 , 杨铁军 , 李纯纯 , 黄琳 . 基于神经网络架构搜索的铭牌目标检测方法[J]. 图学学报, 2023, 44(4): 718-727.
[6]	余伟群, 刘佳涛, 张亚萍. 融合注意力的拉普拉斯金字塔单目深度估计 [J]. 图学学报, 2023, 44(4): 728-738.
[7]	郭印宏, 王立春, 李爽. 基于重复性和特异性约束的图像特征匹配 [J]. 图学学报, 2023, 44(4): 739-746.
[8]	毛爱坤, 刘昕明, 陈文壮, 宋绍楼. 改进YOLOv5算法的变电站仪表目标检测方法[J]. 图学学报, 2023, 44(3): 448-455.
[9]	郝鹏飞, 刘立群, 顾任远. YOLO-RD-Apple果园异源图像遮挡果实检测模型[J]. 图学学报, 2023, 44(3): 456-464.
[10]	罗文宇, 傅明月. 基于YoloX-ECA模型的非法野泳野钓现场监测技术[J]. 图学学报, 2023, 44(3): 465-472.
[11]	杨柳, 吴晓群. 基于深度学习的三维形状补全研究综述[J]. 图学学报, 2023, 44(2): 201-215.
[12]	曾武, 朱恒亮, 邢树礼, 林江宏, 毛国君. 显著性检测引导的图像数据增强方法[J]. 图学学报, 2023, 44(2): 260-270.
[13]	陈刚 , 张培基 , 龚冬冬 , 于俊清 . 火电厂监控视频安全服检测方法研究[J]. 图学学报, 2023, 44(2): 291-297.
[14]	罗启明, 吴昊, 夏信, 袁国武. 基于 Dual Dense U-Net 的云南壁画破损区域预测[J]. 图学学报, 2023, 44(2): 304-312.
[15]	成浪, 敬超. 基于改进 YOLOv7 的 X 线图像旋转目标检测[J]. 图学学报, 2023, 44(2): 324-334.