Fusing prior knowledge reasoning for surface defect detection

doi:10.11996/JG.j.2095-302X.2024050957

Abstract

Abstract:

Current surface defect detection methods based on deep learning mainly focus on the individual identification of defect instances, considering defect detection only from the aspect of region features. However, this method overlooks the high-level relation between defects, which will inevitably lead to defect detection errors. To address the above problems, a surface defect detection network (PKR-Net) based on prior knowledge reasoning was proposed. Specifically, PKR-Net mainly consists of two parts, namely, the explicit knowledge reasoning module (EKRM) and the implicit knowledge reasoning module (IKRM). EKRM constructed an explicit relation graph (ERG) to capture the global co-occurrence relation between defects in the dataset, thereby obtaining co-occurrence relation features. Meanwhile, IKRM constructed an implicit relation graph (IRG) to capture the local spatial relation between defects in the image, thereby obtaining spatial relation features. Finally, the co-occurrence relation features and spatial relation features were fused and re-fed into the classification and regression layers to improve detection performance. Experimental verification was conducted on the industrial defect datasets Textile, NEU-DET and GC10-DET. The experimental results showed that the mAP of the proposed network model improved by 14.8%, 8.2%, and 18.9%, respectively, compared with the baseline model Faster RCNN. Compared with other defect detection models, the proposed model can achieve better detection performance, verifying its effectiveness.

Key words: surface defect detection, prior knowledge, graph convolutional network, object detection, deep learning

CLC Number:

JIANG Xiaoheng, DUAN Jinzhong, LU Yang, CUI Lisha, XU Mingliang. Fusing prior knowledge reasoning for surface defect detection[J]. Journal of Graphics, 2024, 45(5): 957-967.

Figures/Tables 15

Fig. 1 Structure of the PKR-Net model

Fig. 2 Explicit knowledge reasoning module

Fig. 3 Co-occurrence frequency matrix

Fig. 4 Co-occurrence relation matrix

Fig. 5 Implicit knowledge reasoning module

Table 1 Comparison of evaluation results of each model on different defect datasets

模型	主干网络	Textile数据集/%			NEU-DET数据集/%			GC10-DET数据集/%
模型	主干网络	mAP	AP₅₀	AP₇₅	mAP	AP₅₀	AP₇₅	mAP	AP₅₀	AP₇₅
SSD512	VGG16	37.9	71.5	30.9	31.2	67.5	24.3	29.2	59.9	26.7
Faster RCNN	ResNet101	41.2	76.4	34.9	36.6	72.7	32.9	28.5	62.3	23.1
FCOS	ResNet101	39.9	73.4	35.3	37.1	70.2	34.0	30.5	62.3	25.6
Sparse RCNN	ResNet101	39.9	67.9	38.8	38.1	69.0	37.3	28.9	59.7	24.6
RetinaNet	ResNet101	40.7	74.2	34.9	36.0	73.5	30.3	30.0	59.6	28.4
Cascade RCNN	ResNet101	46.2	79.0	44.4	38.7	73.5	36.6	32.5	65.5	30.2
YOLOX-s	CSPDarkNet53	46.5	79.5	41.9	37.7	71.7	34.7	31.3	63.3	29.8
Reasoning RCNN	ResNet101	45.5	78.4	40.7	38.2	73.5	36.6	31.2	65.8	27.2
LF-YOLO	-	39.0	71.2	38.0	33.6	68.3	28.4	31.0	67.0	26.1
CANet	ResNet101	41.1	73.8	39.4	38.2	72.6	32.2	30.6	63.5	26.1
PKR-Net	ResNet101	47.3	80.2	45.6	39.6	76.4	35.6	33.9	67.4	30.7

Fig. 6 Comparison of precision-recall curves of each model on different defect datasets ((a) Textile dataset; (b) NEU-DET dataset; (c) GC10-DET dataset)

Table 2 Comparison of F1 metrics of different models on different datasets /%

模型	Textile 数据集	NEU-DET 数据集	GC10-DET 数据集
SSD512	71.1	68.2	57.5
Faster RCNN	70.0	71.2	58.5
FCOS	69.1	70.5	62.7
Sparse RCNN	64.7	67.6	57.1
RetinaNet	67.0	70.7	61.2
Cascade RCNN	74.0	72.7	62.3
YOLOX-s	76.6	69.7	64.1
Reasoning RCNN	74.5	70.0	60.4
LF-YOLO	67.4	65.4	64.0
CANet	68.6	71.7	62.2
PKR-Net	77.3	74.7	64.8

Table 3 Ablation experiment results of different loss weights on Textile dataset

φ	γ	mAP/%	AP₅₀/%
1.00	1.00	45.6	78.6
1.00	0.75	46.8	79.5
1.00	0.50	47.3	80.2
1.00	0.25	45.5	78.1

Table 4 Ablation experiment results under different modules on Textile dataset

模型	EKRM	IKRM	mAP/%	AP₅₀/%
Faster RCNN	-	-	41.2	76.4
PKR-Net	✓	-	44.6	76.8
PKR-Net	-	✓	45.0	77.0
PKR-Net	✓	✓	47.3	80.2

Table 5 Ablation experiment results under different knowledge on Textile dataset

模块	知识形式	mAP/%
EKRM	全0	41.8
	全1	42.3
	随机	42.1
	正常	44.6
IKRM	全0	41.7
	全1	42.4
	随机	42.3
	正常	45.0

Fig. 7 Comparison of detection errors between baseline and PKR-Net on different defect datasets ((a) Textile dataset; (b) NEU-DET dataset; (c) GC10-DET dataset; (d) Textile dataset; (e) NEU-DET dataset; (f) GC10-DET dataset)

Fig. 8 Comparison of detection results of different methods on Textile dataset ((a) Ground Truth; (b) SSD512; (c) Faster RCNN; (d) FCOS; (e) Sparse RCNN; (f) RetinaNet; (g) Cascade RCNN; (h) YOLOx-s; (i) Reasoning RCNN; (j) LF-YOLO; (k) CANet; (l) PKR-Net)

Fig. 9 Comparison of detection results of different methods on NEU-DET dataset ((a) Ground Truth; (b) SSD512; (c) Faster RCNN; (d) FCOS; (e) Sparse RCNN; (f) RetinaNet; (g) Cascade RCNN; (h) YOLOx-s; (i) Reasoning RCNN; (j) LF-YOLO; (k) CANet; (l) PKR-Net)

Fig. 10 Comparison of detection results of different methods on GC10-DET dataset ((a) Ground Truth; (b) SSD512; (c) Faster RCNN; (d) FCOS; (e) Sparse RCNN; (f) RetinaNet; (g) Cascade RCNN; (h) YOLOx-s; (i) Reasoning RCNN; (j) LF-YOLO; (k) CANet; (l) PKR-Net)

References 44

[1]	王素琴, 任琪, 石敏, 等. 基于异常检测的产品表面缺陷检测与分割[J]. 图学学报, 2022, 43(3): 377-386.
	WANG S Q, REN Q, SHI M, et al. Product surface defect detection and segmentation based on anomaly detection[J]. Journal of Graphics, 2022, 43(3): 377-386 (in Chinese). DOI
[2]	PARK Y, KWEON I S. Ambiguous surface defect image classification of AMOLED displays in smartphones[J]. IEEE Transactions on Industrial Informatics, 2016, 12(2): 597-607.
[3]	ZHANG Y, GAO Y, SHEN L Y. Steel defect detection based on modified RetinaNet[C]// 2022 26th International Conference on Pattern Recognition. New York: IEEE Press, 2022: 3572-3579.
[4]	HE Y, SONG K C, MENG Q G, et al. An end-to-end steel surface defect detection approach via fusing multiple hierarchical features[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(4): 1493-1504.
[5]	王道累, 刘易腾, 杜文斌, 等. 基于级联孪生密集网络的金属表面缺陷检测方法[J]. 计算机辅助设计与图形学学报, 2022, 34(6): 946-952.
	WANG D L, LIU Y T, DU W B, et al. A cascaded twin dense network based metallic surface defect detection algorithm[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(6): 946-952 (in Chinese).
[6]	YU X Y, LYU W T, ZHOU D, et al. ES-Net: efficient scale-aware network for tiny defect detection[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 3511314.
[7]	CUI L S, JIANG X H, XU M L, et al. SDDNet: a fast and accurate network for surface defect detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 2505713.
[8]	罗东亮, 蔡雨萱, 杨子豪, 等. 工业缺陷检测深度学习方法综述[J]. 中国科学: 信息科学, 2022, 52(6): 1002-1039.
	LUO D L, CAI Y X, YANG Z H, et al. Survey on industrial defect detection with deep learning[J]. Scientia Sinica Informationis, 2022, 52(6): 1002-1039 (in Chinese).
[9]	JIANG X H, YAN F, LU Y, et al. Joint attention-guided feature fusion network for saliency detection of surface defects[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 2520912.
[10]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
[11]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2015-04-10) [2024-03-12]. http://arxiv.org/abs/1409.1556.
[12]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 770-778.
[13]	陶显, 侯伟, 徐德. 基于深度学习的表面缺陷检测方法综述[J]. 自动化学报, 2021, 47(5): 1017-1034.
	TAO X, HOU W, XU D. A survey of surface defect detection methods based on deep learning[J]. Acta Automatica Sinica, 2021, 47(5): 1017-1034 (in Chinese).
[14]	胡欣, 胡帅, 马丽军, 等. 基于融合MBAM与YOLOv5的PCB缺陷检测方法[J]. 图学学报, 2024, 45(1): 47-55. DOI
	HU X, HU S, MA L J, et al. PCB defect detection method based on fusion of MBAM and YOLOv5[J]. Journal of Graphics, 2024, 45(1): 47-55 (in Chinese). DOI
[15]	JIA P T, GUO T, GUO F J, et al. Detection model of drainage pipe defect based on improved YOLOv5[C]// 2023 8th International Conference on Intelligent Computing and Signal Processing. New York: IEEE Press, 2023: 1950-1955.
[16]	ZENG Q T, WEI D B, ZHANG X, et al. MFAM-Net: a surface defect detection network for strip steel via multiscale feature fusion and attention mechanism[C]// 2023 International Conference on New Trends in Computational Intelligence. New York: IEEE Press, 2023: 117-121.
[17]	FANG Y, KUAN K, LIN J, et al. Object detection meets knowledge graphs[EB/OL]. [2024-02-23]. https://www.ijcai.org/proceedings/2017/230.
[18]	JIANG C H, XU H, LIANG X D, et al. Hybrid knowledge routed modules for large-scale object detection[C]// The 32nd International Conference on Neural Information Processing Systems. Montréal: Curran Associates Inc., 2018: 1559-1570.
[19]	XU H, JIANG C H, LIANG X D, et al. Reasoning-RCNN: unifying adaptive global reasoning into large-scale object detection[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 6412-6421.
[20]	ZHAI Y J, YANG X, WANG Q M, et al. Hybrid knowledge R-CNN for transmission line multifitting detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 5013312.
[21]	PIECH C, BASSEN J, HUANG J, et al. Deep knowledge tracing[C]// The 28th International Conference on Neural Information Processing Systems. California: MIT Press, 2015: 505-513.
[22]	SCARSELLI F, GORI M, TSOI A C, et al. The graph neural network model[J]. IEEE Transactions on Neural Networks, 2009, 20(1): 61-80. DOI PMID
[23]	KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. (2017-02-22) [2024-01-13]. http://arxiv.org/abs/1609.02907.
[24]	VELIČKOVIĆ P, CUCURULL G, CASANOVA A, et al. Graph attention networks[EB/OL]. (2018-02-04) [2024-02-22]. http://arxiv.org/abs/1710.10903.
[25]	ZHAI Y J, WANG Q M, YANG X, et al. Multi-fitting detection on transmission line based on cascade reasoning graph network[J]. IEEE Transactions on Power Delivery, 2022, 37(6): 4858-4868.
[26]	SHU X, LIU R, XU J. A semantic relation graph reasoning network for object detection[C]// 2021 IEEE 10th Data Driven Control and Learning Systems Conference. New York: IEEE Press, 2021: 1309-1314.
[27]	HWANG J, KANG S, CHUNG K. GCN-based semantic relation network for few-shot object detection[C]// 2022 13th International Conference on Information and Communication Technology Convergence. New York: IEEE Press, 2022: 1689-1693.
[28]	LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 936-944.
[29]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]// The 15th European Conference on Computer Vision. Heidelberg: Springer, 2018: 3-19.
[30]	HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2980-2988.
[31]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7132-7141.
[32]	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. DOI PMID
[33]	LV X M, DUAN F J, JIANG J J, et al. Deep metallic surface defect detection: the new benchmark and detection network[J]. Sensors, 2020, 20(6): 1562.
[34]	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft coco: common objects in context[C]// The 13th European Conference on Computer Vision. Heidelberg: Springer, 2014: 740-755.
[35]	CHEN K, WANG J Q, PANG J M, et al. MMDetection: open MMLab detection toolbox and benchmark[EB/OL]. (2019-06-17) [2024-02-12]. http://arxiv.org/abs/1906.07155.
[36]	RUSSAKOVSKY O, DENG J, SU H, et al. ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115(3): 211-252.
[37]	LIU W, ANGUELOV D, ERHAN D, et al. SSD: Single shot MultiBox detector[C]// The 14th European Conference on Computer Vision. Heidelberg: Springer, 2016: 21-37.
[38]	TIAN Z, SHEN C H, CHEN H, et al. FCOS: fully convolutional one-stage object detection[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 9626-9635.
[39]	SUN P Z, ZHANG R F, JIANG Y, et al. Sparse R-CNN: end-to-end object detection with learnable proposals[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 14449-14458.
[40]	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2999-3007.
[41]	CAI Z W, VASCONCELOS N. Cascade R-CNN: Delving into high quality object detection[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 6154-6162.
[42]	GE Z, LIU S T, WANG F, et al. YOLOX: exceeding YOLO series in 2021[EB/OL]. (2021-08-06) [2024-03-11]. http://arxiv.org/abs/2107.08430.
[43]	LIU M Y, CHEN Y P, XIE J M, et al. LF-YOLO: a lighter and faster YOLO for weld defect detection of X-ray image[J]. IEEE Sensors Journal, 2023, 23(7): 7430-7439.
[44]	HOU X Q, LIU M Q, ZHANG S L, et al. CANet: contextual information and spatial attention based network for detecting small defects in manufacturing industry[J]. Pattern Recognition, 2023, 140: 109558.