Automatic reading of pointer meters based on R-YOLOv7 and MIMO-CTFNet

doi:10.11996/JG.j.2095-302X.2024061313

Abstract

Abstract:

To solve the problems in current pointer meter reading methods, such as the complicated reading process, significant reading errors, and the motion blur caused by camera shakes, an automatic reading method based on R-YOLOv7 and MIMO-CTFNet (multi-input multi-output CNN-transformer fusion network) was proposed. First, the R-YOLOv7 algorithm was constructed to consider both accuracy and lightweight for detecting the dial and its key information. Then, a MIMO-CTFNet algorithm was designed to recover the motion-blurred meter images. Finally, the angle method based on the extracted small scales was utilized to perform meter reading. The experimental results showed that for the data set of dial key information finding, the parameters, FLOPs, ADT, and mAP50:95 were 12 M, 60.30 G, 17.04 ms, and 86.5%, respectively. The PSNR and SSIM of the improved MIMO-CTFNet algorithm achieved 33.05 dB and 0.935 3, respectively. The maximum fiducial error of the proposed reading method was 0.35%, and the reading time for images requiring and not requiring motion blur was 0.561 s and 0.128 s, respectively, validating the effectiveness of the proposed method.

Key words: pointer meters, R-YOLOv7, MIMO-CTFNet, automatic reading, light weight

CLC Number:

TP391
TH70

LI Shengtao, HOU Liqun, DONG Yasong. Automatic reading of pointer meters based on R-YOLOv7 and MIMO-CTFNet[J]. Journal of Graphics, 2024, 45(6): 1313-1327.

Figures/Tables 36

References 31

[1]	熊国良, 肖文明, 王小明. 基于视觉的指针式仪表检测与识别方法综述[J]. 传感器与微系统, 2020, 39(12): 1-3, 9.
	XIONG G L, XIAO W M, WANG X M. Review of pointer meter detection and recognition method based on vision[J]. Transducer and Microsystem Technologies, 2020, 39(12): 1-3, 9. (in Chinese)
[2]	周泓, 徐海儿, 耿晨歌. 基于HSI模型和Hough变换的指针式汽车仪表自动校验[J]. 浙江大学学报(工学版), 2010, 44(6): 1108-1112.
	ZHOU H, XU H E, GENG C G. Automatic checking of pointer automotive dashboard based on HIS model and Hough transformation[J]. Journal of Zhejiang University (Engineering Science), 2010, 44(6): 1108-1112. (in Chinese)
[3]	黄炎, 李文胜, 麦晓明, 等. 基于一维测量线映射的变电站指针仪表智能识读方法[J]. 广东电力, 2018, 31(12): 80-85.
	HUANG Y, LI W S, MAI X M, et al. Intelligent recognition method for substation pointer meter images based on one-dimensional measuring line mapping[J]. Guangdong Electric Power, 2018, 31(12): 80-85. (in Chinese)
[4]	阎光伟, 刘润泽, 焦润海, 等. 基于改进Cascade RCNN的输电线路防振锤脱落检测方法[J]. 图学学报, 2023, 44(5): 849-860. DOI
	YAN G W, LIU R Z, JIAO R H, et al. Detection method of dropped anti-vibration hammer for transmission line based on improved Cascade RCNN[J]. Journal of Graphics, 2023, 44(5): 849-860. (in Chinese)
[5]	万吉林, 王慧芳, 管敏渊, 等. 基于Faster R-CNN和U-Net的变电站指针式仪表读数自动识别方法[J]. 电网技术, 2020, 44(8): 3097-3105.
	WAN J L, WANG H F, GUAN M Y, et al. An automatic identification for reading of substation pointer-type meters using faster R-CNN and U-Net[J]. Power System Technology, 2020, 44(8): 3097-3105. (in Chinese)
[6]	陶金, 林文伟, 曾亮, 等. 基于YOLOv4-tiny和Hourglass的指针式仪表读数识别[J]. 电子测量与仪器学报, 2023, 37(5): 1-10.
	TAO J, LIN W W, ZENG L, et al. Pointer meter reading recognition based on YOLOv4-tiny and Hourglass[J]. Journal of Electronic Measurement and Instrumentation, 2023, 37(5): 1-10. (in Chinese)
[7]	HOU L Q, WANG S, SUN X P, et al. A pointer meter reading recognition method based on YOLOX and semantic segmentation technology[J]. Measurement, 2023, 218: 113241.
[8]	赵振兵, 马迪雅, 石颖, 等. 基于改进YOLOX的变电站仪表外观缺陷检测算法[J]. 图学学报, 2023, 44(5): 937-946. DOI
	ZHAO Z B, MA D Y, SHI Y, et al. Appearance defect detection algorithm of substation instrument based on improved YOLOX[J]. Journal of Graphics, 2023, 44(5): 937-946. (in Chinese)
[9]	UEDA S, SUZUKI K, KANNO J, et al. A two-stage deep learning-based approach for automatic reading of analog meters[C]// The 11th International Conference on Soft Computing and Intelligent Systems and 21st International Symposium on Advanced Intelligent Systems. New York: IEEE Press, 2020: 1-6.
[10]	孙晓朋, 侯立群, 渠怀胜. 基于卷积神经网络的渐进式指针表自动读数方法[J]. 传感技术学报, 2021, 34(10): 1326-1333.
	SUN X P, HOU L Q, QU H S. Gradually Automatic reading method for pointer meters using convolutional neural network[J]. Chinese Journal of Sensors and Actuators, 2021, 34(10): 1326-1333. (in Chinese)
[11]	ZHANG T Y, SUEN C Y. A fast parallel algorithm for thinning digital patterns[J]. Communications of the ACM, 1984, 27(3): 236-239.
[12]	杨武, 胡敏, 常鑫, 等. 改进的DeepLabV3+指针式仪表图像分割算法[J]. 国外电子测量技术, 2024, 43(1): 10-19.
	YANG W, HU M, CHANG X, et al. Improved image segmentation algorithm of DeepLabV3+ pointer meter[J]. Foreign Electronic Measurement Technology, 2024, 43(1): 10-19. (in Chinese)
[13]	朱斌滨, 樊绍胜. 雨雾环境下的变电站指针式仪表识别方法[J]. 激光与光电子学进展, 2021, 58(24): 2410008.
	ZHU B B, FAN S S. Reading method of substation pointer meter in rain-fog environment[J]. Laser & Optoelectronics Progress, 2021, 58(24): 2410008. (in Chinese)
[14]	ZHAI X M, PEI D. Research on intelligent reading algorithm of pointer meter based on deep learning[J]. International Core Journal of Engineering, 2022, 8(4): 554-565.
[15]	侯卓成, 欧阳华, 胡鑫, 等. 基于深度学习的模糊指针式仪表矫正读数方法[J]. 电子测量技术, 2023, 46(9): 158-165.
	HOU Z C, OUYANG H, HU X, et al. Correction reading method of fuzzy pointer instrument based on deep learning[J]. Electronic Measurement Technology, 2023, 46(9): 158-165. (in Chinese)
[16]	MA Y F, JIANG Q, WANG J J, et al. An automatic reading method of pointer instruments[C]// 2017 Chinese Automation Congress. New York: IEEE Press, 2017: 1448-1453.
[17]	HOU L Q, QU H S. Automatic recognition system of pointer meters based on lightweight CNN and WSNs with on-sensor image processing[J]. Measurement, 2021, 183: 109819.
[18]	HOU L Q, SUN X P, WANG S. A coarse-fine reading recognition method for pointer meters based on CNN and computer vision[J]. Engineering Research Express, 2022, 4(3): 035046.
[19]	GAO H J, YI M, YU J Y, et al. Character segmentation-based coarse-fine approach for automobile dashboard detection[J]. IEEE Transactions on Industrial Informatics. 2019, 15(10): 5413-5424.
[20]	LI Z, ZHOU Y S, SHENG Q H, et al. A high-robust automatic reading algorithm of pointer meters based on text detection[J]. Sensors, 2020, 20(20): 5946.
[21]	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 7464-7475.
[22]	YANG X, YAN J C. Arbitrary-oriented object detection with circular smooth label[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 677-694.
[23]	OUYANG D L, HE S, ZHANG G Z, et al. Efficient multi-scale attention module with cross-spatial learning[C]// 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. New York: IEEE Press, 2023: 1-5.
[24]	LEE J, PARK S, MO S, et al. Layer-adaptive sparsity for the magnitude-based pruning[EB/OL]. (2021-05-09) [2024-05-23]. https://arxiv.org/abs/2010.07611.
[25]	YANG X, YANG J R, YAN J C, et al. SCRDeT: towards more robust detection for small, cluttered and rotated objects[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 8231-8240.
[26]	CHO S J, JI S W, HONG J P, et al. Rethinking coarse-to-fine approach in single image deblurring[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 4621-4630.
[27]	ZAMIR S W, ARORA A, KHAN S, et al. Restormer: efficient transformer for high-resolution image restoration[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 5718-5729.
[28]	NEWELL A, YANG K Y, DENG J. Stacked hourglass networks for human pose estimation[C]// The 14th European Conference on Computer Vision. Cham: Springer, 2016: 483-499.
[29]	LIU Y C, SHAO Z R, TENG Y Y, et al. NAM: normalization-based attention module[EB/OL]. (2021-11-24) [2024-05-23]. https://arxiv.org/abs/2111.12419.
[30]	CUI Y N, TAO Y, BING Z S, et al. Selective frequency network for image restoration[EB/OL]. [2024-05-23]. https://openreview.net/forum?id=tyZ1ChGZIKO.
[31]	半片青柠. 工业场景下指针表数据集_语义分割做指针识别资源-CSDN文库[EB/OL]. (2024-05-11) [2024-05-23]. https://download.csdn.net/download/sinat_40587853/87453496?
	Half a Slice of Lime. Pointer table dataset for industrial scenarios_semantic segmentation for pointer recognition resources - CSDN library[EB/OL]. (2024-05-11) [2024-05-23]. https://download.csdn.net/download/sinat_40587853/87453496?. (in Chinese)

参数	配置
操作系统	Ubuntu20.04
CPU型号	Intel(R) Xeon(R) Platinum 8255C
显卡(GPU)型号	NVIDIA GeForce RTX2080Ti
显卡内存	11 G
深度学习框架	Pytorch 1.10.0
编程语言	Python 3.8

参数	配置
操作系统	Ubuntu20.04
CPU型号	Intel(R) Xeon(R) Platinum 8255C
显卡(GPU)型号	NVIDIA GeForce RTX2080Ti
显卡内存	11 G
深度学习框架	Pytorch 1.10.0
编程语言	Python 3.8

算法	参数量/ M	FLOPs/ G	ADT/ ms	PSNR/ dB	SSIM
Baseline	6.8	63.75	30.35	30.63	0.924 7
Baseline+A	12.3	104.18	126.45	31.81	0.931 5
Baseline+B	3.8	36.42	106.87	32.49	0.930 4
Baseline+A+B	9.3	76.85	251.78	33.05	0.935 3

算法	参数量/ M	FLOPs/ G	ADT/ ms	PSNR/ dB	SSIM
Baseline	6.8	63.75	30.35	30.63	0.924 7
Baseline+A	12.3	104.18	126.45	31.81	0.931 5
Baseline+B	3.8	36.42	106.87	32.49	0.930 4
Baseline+A+B	9.3	76.85	251.78	33.05	0.935 3

算法	参数量/ M	FLOPs/ G	ADT/ ms	PSNR/ dB	SSIM
MIMO-UNet	6.80	63.75	30.35	30.63	0.924 7
MIMO-UNet+	16.10	150.72	48.82	30.95	0.930 8
SFNet	13.27	125.43	252.18	31.09	0.934 7
Ours	9.30	76.85	251.78	33.05	0.935 3