Defect detection method of transmission line bolts based on DETR and prior knowledge fusion

doi:10.11996/JG.j.2095-302X.2023030438

Abstract

Abstract:

In order to address the problem of deep learning model unable to learn the prior knowledge of bolt targets, difficulty in locating its defects quickly and accurately only through visual features, and the limited number and unbalanced categories of bolt defect samples, this paper proposed the method of incorporating the deep learning model and the prior knowledge of bolts. DETR (detection transformer) was selected as the baseline model, and an improved DETR model was designed and implemented by incorporating DETR and prior knowledge. First, the visual-knowledge attention module was used to fuse the visual features of the bolt image with the prior knowledge of the bolt, generating the enhanced visual features corresponding to the bolts. Then, the enhanced visual features were sent to the DETR model framework, which was based on the Transformer encoding-decoding structure, thus identifying and classifying bolt targets. Finally, to overcome the problem of few and unbalanced samples of bolt critical defects, a class incremental learning loss function (CILLF) was introduced to enhance the identification ability of the model and alleviate the long-tail distribution problem of bolt defect samples. The simulation results demonstrated that the improved DETR model achieved an increase of 2.8 percentage points in mAP on the transmission line bolt defect sample compared with the baseline model DETR. Compared with the mainstream Faster R-CNN and YOLOv5l models, the improved DETR model showed significant improvement in detecting category images with few bolt defect samples under the long-tail distribution.

Key words: bolt defect detection, Transformer, DETR, prior knowledge, augmented visual features, incremental learning-like loss function

CLC Number:

TP391

LI Gang, ZHANG Yun-tao, WANG Wen-kai, ZHANG Dong-yang. Defect detection method of transmission line bolts based on DETR and prior knowledge fusion[J]. Journal of Graphics, 2023, 44(3): 438-447.

Figures/Tables 15

Fig. 1 DETR model structure

Table 1 The average detection rate of the model under different learning rates and iterations (%)

迭代次数	学习率
迭代次数	0.01	0.001	0.000 5	0.000 1
1 000	46.73	45.39	45.23	44.85
2 000	48.65	45.63	48.14	46.32
4 000	49.11	49.17	49.96	47.64
6 000	49.30	48.96	49.96	48.59

Table 2 The average precision and average precision of each category corresponding to different Mini-batch parameters (%)

Mini-batch	AP
Mini-batch	正常螺栓	缺销钉	缺螺母	缺垫片	AP_0.5
8	71.6	59.6	53.6	55.6	60.1
16	75.0	61.4	54.1	58.7	62.3
32	74.7	64.0	54.5	60.4	63.4
64	78.6	57.6	54.7	61.1	63.0
128	83.3	51.1	54.2	61.8	62.6

Table 3 Average detection rate of models corresponding to different Encoder and Decoder parameters (%)

Encoder参数	平均检出率
Encoder参数	Decoder参数1	Decoder参数2	Decoder参数4	Decoder参数6	Decoder参数8	Decoder参数10
2	41.1	44.6	46.5	50.2	53.1	52.2
4	42.3	47.5	49.2	51.9	54.7	52.6
6	44.5	47.9	51.8	54.3	55.6	53.2
8	44.9	48.7	53.1	52.6	54.8	51.0
10	44.6	50.4	53.5	54.0	52.4	49.9

Fig. 2 Bolt defect detection framework based on fusion of DETR and prior knowledge

Fig. 3 Vision-knowledge attention module

Fig. 4 High-definition aerial image

Table 4 Statistical table of the number of samples of various types of bolts on different fittings

典型金具	正常	缺垫片	缺螺母	缺销钉
挂点	1349	312	41	109
防震锤	210	217	12	37
绝缘子	356	112	26	75
间隔棒	145	15	6	4

Table 5 The composition of the components in the normal state of the bolts on different fittings

金具部件位置	正常状态下螺栓元件组成
挂点	螺母+销钉或垫片+销钉
防震锤	螺母+销钉或垫片+销钉
绝缘子	螺母+销钉
间隔棒	垫片+螺母+销钉或垫片+销钉

Table 6 Partial bolt defect image and corresponding label example

图像	先验知识	标签信息
	挂点主视图	ls-zc
	防震锤俯视图	ls-qdp
	挂点俯视图	ls-qlm
	间隔棒主视图	ls-qxd

Fig. 5 Comparison of the detection effect of the model before and after the fusion of prior knowledge ((a) DETR; (b) DETR+ V-K Attention)

Fig. 6 The detection effect of the model before and after the application of the class incremental loss function ((a) DETR; (b) DETR+ CILLF)

Table 7 Comparison results of different models

方法	mAP (%)	FPS (s)	Weights (MB)
YOLOv5l	57.9	25.7	87.33
Faster R-CNN	65.4	6.9	141.47
DETR	67.8	7.6	126.07
改进DETR	70.6	5.5	174.72

Table 8 Comprehensive use of different improvement strategies to compare the detection effects

组别	方法	mAP (%)	FPS (s)
1	DETR	67.8	7.6
2	+ CILLF	69.3	7.1
3	+ CILLF +V-K Attention	70.6	5.5

Fig. 7 Comparison of detection results between mainstream target detection algorithms and improved DETR model ((a) Faster R-CNN; (b) YOLOv5l; (c) Improved DETR)

References 21

[1]	赵振兵, 张薇, 翟永杰, 等. 电力视觉技术的概念、研究现状与展望[J]. 电力科学与工程, 2020, 36(1): 1-8.
	ZHAO Z B, ZHANG W, ZHAI Y J, et al. Concept, research status and prospect of electric power vision technology[J]. Electric Power Science and Engineering, 2020, 36(1): 1-8. (in Chinese)
[2]	ZHOU X Y, ZHUO J C, KRÄHENBÜHL P. Bottom-up object detection by grouping extreme and center points[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 850-859.
[3]	孙宗康, 饶睦敏, 曹裕灵, 等. 基于小样本不均衡数据的供水管道泄漏智能检测算法[J]. 图学学报, 2022, 43(5): 825-831.
	SUN Z K, RAO M M, CAO Y L, et al. Water supply pipeline leakage intelligent detection algorithm based on small and unbalanced data[J]. Journal of Graphics, 2022, 43(5): 825-831. (in Chinese)
[4]	奚川龙. 基于路侧毫米波雷达和相机信息融合的车辆目标检测研究[D]. 重庆: 重庆邮电大学, 2021.
	XI C L. Research on vehicle target detection based on information fusion of roadside millimeter wave radar and camera[D]. Chongqing: Chongqing University of Posts and Telecommunications, 2021. (in Chinese)
[5]	赵振兵, 蒋志钢, 李延旭, 等. 输电线路部件视觉缺陷检测综述[J]. 中国图象图形学报, 2021, 26(11): 2545-2560.
	ZHAO Z B, JIANG Z G, LI Y X, et al. Overview of visual defect detection of transmission line components[J]. Journal of Image and Graphics, 2021, 26(11): 2545-2560. (in Chinese)
[6]	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 779-788.
[7]	谷雨, 赵军. 列车闸瓦钎及闸瓦故障图像检测算法研究[J/OL]. (2022-07-20) [2022-08-08]. http://kns.cnki.net/kcms/det-ail/10.1034.T.20220719.1607.002.html.
	GU Y, ZHAO J. Research on image detection algorithm of freight train brake shoe bolt and brake shoe fault[J/OL]. (2022-07-20) [2022-08-08]. http://kns.cnki.net/kcms/det-ail/10.1034.T.20220719.1607.002.html. (in Chinese)
[8]	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot MultiBox detector[M]//Computer Vision - ECCV 2016. Cham: Springer International Publishing, 2016: 21-37.
[9]	YI J R, WU P X, METAXAS D N. ASSD: attentive single shot multibox detector[J]. Computer Vision and Image Understanding, 2019, 189: 102827. DOI URL
[10]	赵振兵, 李延旭, 甄珍, 等. 结合KL散度和形状约束的Faster R-CNN典型金具检测方法[J]. 高电压技术, 2020, 46(9): 3018-3026.
	ZHAO Z B, LI Y X, ZHEN Z, et al. Typical fittings detection method with faster R-CNN combining KL divergence and shape constraints[J]. High Voltage Engineering, 2020, 46(9): 3018-3026. (in Chinese)
[11]	翟永杰, 杨旭, 赵振兵, 等. 融合共现推理的Faster R-CNN输电线路金具检测[J]. 智能系统学报, 2021, 16(2): 237-246.
	ZHAI Y J, YANG X, ZHAO Z B, et al. Integrating co-occurrence reasoning for Faster R-CNN transmission line fitting detection[J]. CAAI Transactions on Intelligent Systems, 2021, 16(2): 237-246. (in Chinese)
[12]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all You need[C]// The 31st International Conference on Neural Information Processing Systems. New York: ACM, 2017: 6000-6010.
[13]	ZHU X Z, SU W J, LU L W, et al. Deformable DETR: deformable transformers for end-to-end object detection[EB/OL]. [2022-07-15]. https://arxiv.org/abs/2010.04159.
[14]	张乃雪, 钟羽中, 赵涛, 等. 基于Smooth-DETR的产品表面小尺寸缺陷检测算法[J]. 计算机应用研究, 2022, 39(8): 2520-2525.
	ZHANG N X, ZHONG Y Z, ZHAO T, et al. Detection method for small-size defects based on Smooth-DETR[J]. Application Research of Computers, 2022, 39(8): 2520-2525. (in Chinese)
[15]	熊怡梦. 基于标签语义和Transformer的元学习小样本目标检测方法研究[D]. 西安: 西安电子科技大学, 2021.
	XIONG Y M. Label semantics and Transformer for meta learning few-shot object detection[D]. Xi′an: Xidian University, 2021. (in Chinese)
[16]	LI F, ZHANG H, LIU S L, et al. DN-DETR: accelerate DETR training by introducing query DeNoising[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 13609-13617.
[17]	CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[M]//Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 213-229.
[18]	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2022: 9992-10002.
[19]	孟晓娟, 张月琴, 郝晓丽, 等. 多分类深度卷积生成对抗网络的皮带撕裂检测[J]. 计算机工程与应用, 2021, 57(16): 269-275. DOI
	MENG X J, ZHANG Y Q, HAO X L, et al. Multi-class deep convolutional generative adversarial networks for belt tear detection[J]. Computer Engineering and Applications, 2021, 57(16): 269-275. (in Chinese) DOI
[20]	李军, 李明, 曾蒸, 等. 基于改进激活函数和学习率的双向LSTM研究[J]. 重庆师范大学学报: 自然科学版, 2021, 38(2): 70-76.
	LI J, LI M, ZENG Z, et al. Research on bidirectional LSTM based on improved activation function and learning rate[J]. Journal of Chongqing Normal University: Natural Science, 2021, 38(2): 70-76. (in Chinese)
[21]	何颖宣. 基于多标签学习的螺栓多属性分类方法研究[D]. 北京: 华北电力大学, 2021.
	HE Y X. Research on bolt multi-attribute classification method based on multi-label learning[D]. Beijing: North China Electric Power University, 2021. (in Chinese)