采用DETR与先验知识融合的输电线路螺栓缺陷检测方法

doi:10.11996/JG.j.2095-302X.2023030438

图学学报 ›› 2023, Vol. 44 ›› Issue (3): 438-447.DOI: 10.11996/JG.j.2095-302X.2023030438

• 图像处理与计算机视觉 • 上一篇下一篇

采用DETR与先验知识融合的输电线路螺栓缺陷检测方法

李刚¹^,²(), 张运涛¹, 汪文凯¹, 张东阳¹()

1.华北电力大学计算机系，河北保定 071003
2.复杂能源系统智能计算教育部工程研究中心，河北保定 071003

收稿日期:2022-08-12 接受日期:2022-12-28 出版日期:2023-06-30 发布日期:2023-06-30
通讯作者: 张东阳(1981-)，男，工程师，硕士。主要研究方向为计算机视觉。E-mail：zhdy@ncepu.edu.cn
作者简介:
李刚(1980-)，男，副教授，博士。主要研究方向为故障预测与健康管理等。E-mail：ququ_er2003@126.com
基金资助:
国家自然科学基金项目(51407076);中央高校基本科研业务费专项资金项目(2020MS119)

Defect detection method of transmission line bolts based on DETR and prior knowledge fusion

LI Gang¹^,²(), ZHANG Yun-tao¹, WANG Wen-kai¹, ZHANG Dong-yang¹()

1. Department of Computer, North China Electric Power University, Baoding Hebei 071003, China
2. Engineering Research Center of Intelligent Computing for Complex Energy Systems, Ministry of Education, Baoding Hebei 071003, China

Received:2022-08-12 Accepted:2022-12-28 Online:2023-06-30 Published:2023-06-30
Contact: ZHANG Dong-yang (1981-), engineer, master. His main research interest covers computer vision. E-mail：zhdy@ncepu.edu.cn
About author:
LI Gang (1980-), associate professor, Ph.D. His main research interests cover prognostics and health management, etc. E-mail：ququ_er2003@126.com
Supported by:
National Natural Science Foundation of China(51407076);Fundamental Research Funds for the Central Universities(2020MS119)

摘要/Abstract

摘要：

为了解决深度学习模型无法学习螺栓目标的先验知识、仅通过视觉特征难以快速准确定位其缺陷以及螺栓缺陷样本数量有限、类别不平衡的问题，提出了将深度学习模型与螺栓先验知识相结合的方法。选取端到端目标检测(DETR)为基线模型，设计并实现了一种采用DETR与先验知识融合的改进DETR模型。首先，利用视觉-知识注意力模块将螺栓图像的视觉特征与螺栓先验知识有机融合，获得螺栓对应的增强视觉特征；然后，将增强视觉特征送入基于Transformer编码-解码结构的DETR模型框架中对螺栓目标进行识别与分类；最后，针对螺栓危急缺陷样本少及样本不平衡的问题，引入类增量学习损失函数(CILLF)来增强模型的鉴别能力，缓解螺栓缺陷样本间长尾分布问题。仿真实验结果表明：改进DETR模型在输电线路螺栓缺陷样本上的mAP相较于基线模型DETR提升了2.8个百分点；相较于主流Faster R-CNN，YOLOv5l模型，改进DETR模型在长尾分布下螺栓缺陷样本少的类别图像上的检测效果提升尤为显著。

关键词: 螺栓缺陷检测, Transformer, DETR, 先验知识, 增强视觉特征, 类增量学习损失函数

Abstract:

In order to address the problem of deep learning model unable to learn the prior knowledge of bolt targets, difficulty in locating its defects quickly and accurately only through visual features, and the limited number and unbalanced categories of bolt defect samples, this paper proposed the method of incorporating the deep learning model and the prior knowledge of bolts. DETR (detection transformer) was selected as the baseline model, and an improved DETR model was designed and implemented by incorporating DETR and prior knowledge. First, the visual-knowledge attention module was used to fuse the visual features of the bolt image with the prior knowledge of the bolt, generating the enhanced visual features corresponding to the bolts. Then, the enhanced visual features were sent to the DETR model framework, which was based on the Transformer encoding-decoding structure, thus identifying and classifying bolt targets. Finally, to overcome the problem of few and unbalanced samples of bolt critical defects, a class incremental learning loss function (CILLF) was introduced to enhance the identification ability of the model and alleviate the long-tail distribution problem of bolt defect samples. The simulation results demonstrated that the improved DETR model achieved an increase of 2.8 percentage points in mAP on the transmission line bolt defect sample compared with the baseline model DETR. Compared with the mainstream Faster R-CNN and YOLOv5l models, the improved DETR model showed significant improvement in detecting category images with few bolt defect samples under the long-tail distribution.

Key words: bolt defect detection, Transformer, DETR, prior knowledge, augmented visual features, incremental learning-like loss function

中图分类号:

TP391

李刚, 张运涛, 汪文凯, 张东阳. 采用DETR与先验知识融合的输电线路螺栓缺陷检测方法[J]. 图学学报, 2023, 44(3): 438-447.

LI Gang, ZHANG Yun-tao, WANG Wen-kai, ZHANG Dong-yang. Defect detection method of transmission line bolts based on DETR and prior knowledge fusion[J]. Journal of Graphics, 2023, 44(3): 438-447.

图/表 15

图1 DETR模型结构

Fig. 1 DETR model structure

表1 不同学习率和迭代次数下模型的平均检出率(%)

Table 1 The average detection rate of the model under different learning rates and iterations (%)

迭代次数	学习率
迭代次数	0.01	0.001	0.000 5	0.000 1
1 000	46.73	45.39	45.23	44.85
2 000	48.65	45.63	48.14	46.32
4 000	49.11	49.17	49.96	47.64
6 000	49.30	48.96	49.96	48.59

表2 不同Mini-batch参数对应的各类别的平均精度与平均精度均值(%)

Table 2 The average precision and average precision of each category corresponding to different Mini-batch parameters (%)

Mini-batch	AP
Mini-batch	正常螺栓	缺销钉	缺螺母	缺垫片	AP_0.5
8	71.6	59.6	53.6	55.6	60.1
16	75.0	61.4	54.1	58.7	62.3
32	74.7	64.0	54.5	60.4	63.4
64	78.6	57.6	54.7	61.1	63.0
128	83.3	51.1	54.2	61.8	62.6

表3 不同Encoder和Decoder参数对应的模型的平均检出率(%)

Table 3 Average detection rate of models corresponding to different Encoder and Decoder parameters (%)

Encoder参数	平均检出率
Encoder参数	Decoder参数1	Decoder参数2	Decoder参数4	Decoder参数6	Decoder参数8	Decoder参数10
2	41.1	44.6	46.5	50.2	53.1	52.2
4	42.3	47.5	49.2	51.9	54.7	52.6
6	44.5	47.9	51.8	54.3	55.6	53.2
8	44.9	48.7	53.1	52.6	54.8	51.0
10	44.6	50.4	53.5	54.0	52.4	49.9

图2 基于DETR与先验知识融合的螺栓缺陷检测框架

Fig. 2 Bolt defect detection framework based on fusion of DETR and prior knowledge

图3 视觉-知识注意力模块

Fig. 3 Vision-knowledge attention module

图4 高清航拍图像

Fig. 4 High-definition aerial image

表4 不同金具上各类螺栓样本数量统计表

Table 4 Statistical table of the number of samples of various types of bolts on different fittings

典型金具	正常	缺垫片	缺螺母	缺销钉
挂点	1349	312	41	109
防震锤	210	217	12	37
绝缘子	356	112	26	75
间隔棒	145	15	6	4

表5 不同金具上螺栓正常状态下元件组成情况

Table 5 The composition of the components in the normal state of the bolts on different fittings

金具部件位置	正常状态下螺栓元件组成
挂点	螺母+销钉或垫片+销钉
防震锤	螺母+销钉或垫片+销钉
绝缘子	螺母+销钉
间隔棒	垫片+螺母+销钉或垫片+销钉

表6 部分螺栓缺陷图像和对应的标签示例

Table 6 Partial bolt defect image and corresponding label example

图像	先验知识	标签信息
	挂点主视图	ls-zc
	防震锤俯视图	ls-qdp
	挂点俯视图	ls-qlm
	间隔棒主视图	ls-qxd

图5 融合先验知识前后模型的检测效果对比图

Fig. 5 Comparison of the detection effect of the model before and after the fusion of prior knowledge ((a) DETR; (b) DETR+ V-K Attention)

图6 类增量损失函数应用前后模型的检测效果

Fig. 6 The detection effect of the model before and after the application of the class incremental loss function ((a) DETR; (b) DETR+ CILLF)

表7 不同模型的对比结果

Table 7 Comparison results of different models

方法	mAP (%)	FPS (s)	Weights (MB)
YOLOv5l	57.9	25.7	87.33
Faster R-CNN	65.4	6.9	141.47
DETR	67.8	7.6	126.07
改进DETR	70.6	5.5	174.72

表8 综合使用不同改进策略检测效果对比

Table 8 Comprehensive use of different improvement strategies to compare the detection effects

组别	方法	mAP (%)	FPS (s)
1	DETR	67.8	7.6
2	+ CILLF	69.3	7.1
3	+ CILLF +V-K Attention	70.6	5.5

图7 主流目标检测算法与改进DETR模型检测结果对比((a) Faster R-CNN；(b) YOLOv5l；(c)改进DETR)

Fig. 7 Comparison of detection results between mainstream target detection algorithms and improved DETR model ((a) Faster R-CNN; (b) YOLOv5l; (c) Improved DETR)

参考文献 21

[1]	赵振兵, 张薇, 翟永杰, 等. 电力视觉技术的概念、研究现状与展望[J]. 电力科学与工程, 2020, 36(1): 1-8.
	ZHAO Z B, ZHANG W, ZHAI Y J, et al. Concept, research status and prospect of electric power vision technology[J]. Electric Power Science and Engineering, 2020, 36(1): 1-8. (in Chinese)
[2]	ZHOU X Y, ZHUO J C, KRÄHENBÜHL P. Bottom-up object detection by grouping extreme and center points[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 850-859.
[3]	孙宗康, 饶睦敏, 曹裕灵, 等. 基于小样本不均衡数据的供水管道泄漏智能检测算法[J]. 图学学报, 2022, 43(5): 825-831.
	SUN Z K, RAO M M, CAO Y L, et al. Water supply pipeline leakage intelligent detection algorithm based on small and unbalanced data[J]. Journal of Graphics, 2022, 43(5): 825-831. (in Chinese)
[4]	奚川龙. 基于路侧毫米波雷达和相机信息融合的车辆目标检测研究[D]. 重庆: 重庆邮电大学, 2021.
	XI C L. Research on vehicle target detection based on information fusion of roadside millimeter wave radar and camera[D]. Chongqing: Chongqing University of Posts and Telecommunications, 2021. (in Chinese)
[5]	赵振兵, 蒋志钢, 李延旭, 等. 输电线路部件视觉缺陷检测综述[J]. 中国图象图形学报, 2021, 26(11): 2545-2560.
	ZHAO Z B, JIANG Z G, LI Y X, et al. Overview of visual defect detection of transmission line components[J]. Journal of Image and Graphics, 2021, 26(11): 2545-2560. (in Chinese)
[6]	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 779-788.
[7]	谷雨, 赵军. 列车闸瓦钎及闸瓦故障图像检测算法研究[J/OL]. (2022-07-20) [2022-08-08]. http://kns.cnki.net/kcms/det-ail/10.1034.T.20220719.1607.002.html.
	GU Y, ZHAO J. Research on image detection algorithm of freight train brake shoe bolt and brake shoe fault[J/OL]. (2022-07-20) [2022-08-08]. http://kns.cnki.net/kcms/det-ail/10.1034.T.20220719.1607.002.html. (in Chinese)
[8]	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot MultiBox detector[M]//Computer Vision - ECCV 2016. Cham: Springer International Publishing, 2016: 21-37.
[9]	YI J R, WU P X, METAXAS D N. ASSD: attentive single shot multibox detector[J]. Computer Vision and Image Understanding, 2019, 189: 102827. DOI URL
[10]	赵振兵, 李延旭, 甄珍, 等. 结合KL散度和形状约束的Faster R-CNN典型金具检测方法[J]. 高电压技术, 2020, 46(9): 3018-3026.
	ZHAO Z B, LI Y X, ZHEN Z, et al. Typical fittings detection method with faster R-CNN combining KL divergence and shape constraints[J]. High Voltage Engineering, 2020, 46(9): 3018-3026. (in Chinese)
[11]	翟永杰, 杨旭, 赵振兵, 等. 融合共现推理的Faster R-CNN输电线路金具检测[J]. 智能系统学报, 2021, 16(2): 237-246.
	ZHAI Y J, YANG X, ZHAO Z B, et al. Integrating co-occurrence reasoning for Faster R-CNN transmission line fitting detection[J]. CAAI Transactions on Intelligent Systems, 2021, 16(2): 237-246. (in Chinese)
[12]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all You need[C]// The 31st International Conference on Neural Information Processing Systems. New York: ACM, 2017: 6000-6010.
[13]	ZHU X Z, SU W J, LU L W, et al. Deformable DETR: deformable transformers for end-to-end object detection[EB/OL]. [2022-07-15]. https://arxiv.org/abs/2010.04159.
[14]	张乃雪, 钟羽中, 赵涛, 等. 基于Smooth-DETR的产品表面小尺寸缺陷检测算法[J]. 计算机应用研究, 2022, 39(8): 2520-2525.
	ZHANG N X, ZHONG Y Z, ZHAO T, et al. Detection method for small-size defects based on Smooth-DETR[J]. Application Research of Computers, 2022, 39(8): 2520-2525. (in Chinese)
[15]	熊怡梦. 基于标签语义和Transformer的元学习小样本目标检测方法研究[D]. 西安: 西安电子科技大学, 2021.
	XIONG Y M. Label semantics and Transformer for meta learning few-shot object detection[D]. Xi′an: Xidian University, 2021. (in Chinese)
[16]	LI F, ZHANG H, LIU S L, et al. DN-DETR: accelerate DETR training by introducing query DeNoising[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 13609-13617.
[17]	CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[M]//Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 213-229.
[18]	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2022: 9992-10002.
[19]	孟晓娟, 张月琴, 郝晓丽, 等. 多分类深度卷积生成对抗网络的皮带撕裂检测[J]. 计算机工程与应用, 2021, 57(16): 269-275. DOI
	MENG X J, ZHANG Y Q, HAO X L, et al. Multi-class deep convolutional generative adversarial networks for belt tear detection[J]. Computer Engineering and Applications, 2021, 57(16): 269-275. (in Chinese) DOI
[20]	李军, 李明, 曾蒸, 等. 基于改进激活函数和学习率的双向LSTM研究[J]. 重庆师范大学学报: 自然科学版, 2021, 38(2): 70-76.
	LI J, LI M, ZENG Z, et al. Research on bidirectional LSTM based on improved activation function and learning rate[J]. Journal of Chongqing Normal University: Natural Science, 2021, 38(2): 70-76. (in Chinese)
[21]	何颖宣. 基于多标签学习的螺栓多属性分类方法研究[D]. 北京: 华北电力大学, 2021.
	HE Y X. Research on bolt multi-attribute classification method based on multi-label learning[D]. Beijing: North China Electric Power University, 2021. (in Chinese)

采用DETR与先验知识融合的输电线路螺栓缺陷检测方法

Defect detection method of transmission line bolts based on DETR and prior knowledge fusion

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 15

参考文献 21

相关文章 4

编辑推荐

Metrics

本文评价

[1]	郝帅, 赵新生, 马旭, 张旭, 何田, 侯李祥. 基于 TR-YOLOv5 的输电线路多类缺陷目标检测方法 [J]. 图学学报, 2023, 44(4): 667-676.
[2]	潘东辉, 金映含, 孙旭, 刘玉生, 张东亮. CTH-Net：从线稿和颜色点生成服装图像的 CNN-Transformer 混合网络 [J]. 图学学报, 2023, 44(1): 120-130.
[3]	王玉萍 , 曾毅 , 李胜辉 , 张磊 . 一种基于 Transformer 的三维人体姿态估计方法[J]. 图学学报, 2023, 44(1): 139-145.
[4]	胡海涛 , 杜昊晨 , 王素琴 , 石敏 , 朱登明 , . 改进 YOLOX 的药品泡罩铝箔表面缺陷检测方法[J]. 图学学报, 2022, 43(5): 803-814.