Knee cysts detection algorithm based on Mask R-CNN integrating global-local attention module

doi:10.11996/JG.j.2095-302X.2023061183

Abstract

Abstract:

Accurately detecting knee cysts is an effective means for facilitating early diagnosis and treatment of various knee-related diseases. However, the task of detecting knee cysts can be challenging due to their imaging features' similarity to other lesions in MR imaging, such as intra-articular effusion and cystic tumors. Therefore, a Mask R-CNN multi-task learning model incorporating global-local attention modules was proposed to simultaneously implement the automatic recognition, detection, and segmentation of knee cysts in MRI. Firstly, the method utilized the channel attention mechanism to achieve the weighted fusion of global and local features of knee images, forming a feature map with multi-scale information. This map provided more accurate discriminative features for the model. Secondly, a multi-task uncertainty loss function was introduced, which employed homoskedasticity uncertainty to indicate the relative confidence of each task. It adaptively adjusted the task weights, and automatically searched for the optimal solution. Finally, the GrabCut method was utilized to generate masks based on pre-labeled bounding boxes to further construct knee MRI datasets, enhancing the quality and efficiency of data annotation. The experimental results demonstrated that the proposed method could accurately identify cystic knee lesions in MRI, with an average accuracy of 92.3% for detection and 92.8% for segmentation. These results outperformed other comparison methods in terms of effectiveness.

Key words: knee cysts, multi-task learning, attention mechanism, multi-scale feature fusion, Mask R-CNN

CLC Number:

TP391

ZHANG Li-yuan, ZHAO Hai-rong, HE Wei, TANG Xiong-feng. Knee cysts detection algorithm based on Mask R-CNN integrating global-local attention module[J]. Journal of Graphics, 2023, 44(6): 1183-1190.

Figures/Tables 10

References 24

[1]	DAINESE P, WYNGAERT K V, DE MITS S, et al. Association between knee inflammation and knee pain in patients with knee osteoarthritis: a systematic review[J]. Osteoarthritis and Cartilage, 2022, 30(4): 516-534. DOI URL
[2]	PERRY T A, O'NEILL T W, TOLSTYKH I, et al. Magnetic resonance imaging-assessed subchondral cysts and incident knee pain and knee osteoarthritis: data from the multicenter osteoarthritis study[J]. Arthritis & Rheumatology, 2022, 74(1): 60-69.
[3]	艾阳平. 100例膝关节囊肿的高频超声与MRI分析[J]. 影像研究与医学应用, 2021, 5(10): 107-108.
	AI Y P. High frequency ultrasound and MRI analysis of 100 cases of knee joint cyst[J]. Journal of Imaging Research and Medical Applications, 2021, 5(10): 107-108 (in Chinese).
[4]	DAINESE P, WYNGAERT K V, DE MITS S, et al. Association between knee inflammation and knee pain in patients with knee osteoarthritis: a systematic review[J]. Osteoarthritis and Cartilage, 2022, 30(4): 516-534. DOI URL
[5]	DING Y, ACOSTA R, ENGUIX V, et al. Using deep convolutional neural networks for neonatal brain image segmentation[J]. Frontiers in Neuroscience, 2020, 14: 1-10. DOI URL
[6]	MEHTA D, GAL Y, BOWLING A, et al. Out-of-distribution detection for long-tailed and fine-grained skin lesion images[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2022: 732-742.
[7]	KHAN S, AZAM B, YAO Y C, et al. Deep collaborative network with alpha matte for precise knee tissue segmentation from MRI[J]. Computer Methods and Programs in Biomedicine, 2022, 222: 106963. DOI URL
[8]	AWAN M J, RAHIM M S M, SALIM N, et al. Efficient detection of knee anterior cruciate ligament from magnetic resonance imaging using deep learning approach[J]. Diagnostics, 2021, 11(1): 105. DOI URL
[9]	HUO X Z, SUN G, TIAN S W, et al. HiFuse: hierarchical multi-scale feature fusion network for medical image classification[EB/OL]. (2022-09-21) [2023-01-18]. https://arxiv.org/abs/2209.10218.pdf.
[10]	LIU S S, WANG H L, LI Y, et al. AHU-MultiNet: adaptive loss balancing based on homoscedastic uncertainty in multi-task medical image segmentation network[J]. Computers in Biology and Medicine, 2022, 150: 106157. DOI URL
[11]	HAQUE A, IMRAN A A Z, WANG A, et al. Generalized multi-task learning from substantially unlabeled multi-source medical image data[EB/OL]. (2021-10-25) [2023-01-18]. https://arxiv.org/abs/2110.13185.pdf.
[12]	TIAN Y N, YANG G D, WANG Z, et al. Apple detection during different growth stages in orchards using the improved YOLO-V3 model[J]. Computers and Electronics in Agriculture, 2019, 157: 417-426. DOI URL
[13]	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 7464-7475.
[14]	ZHOU X Y, WANG D Q, KRÄHENBÜHL P. Objects as points[EB/OL]. (2019-04-16) [2023-01-18]. https://arxiv.org/abs/1904.07850.pdf.
[15]	CHEN Y H, WANG H R, LI W, et al. Scale-aware domain adaptive faster R-CNN[J]. International Journal of Computer Vision, 2021, 129(7): 2223-2243. DOI
[16]	XU B B, WANG W S, FALZON G, et al. Automated cattle counting using Mask R-CNN in quadcopter vision system[J]. Computers and Electronics in Agriculture, 2020, 171: 105300. DOI URL
[17]	CAI Z W, VASCONCELOS N. Cascade R-CNN: high quality object detection and instance segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 43(5): 1483-1498. DOI URL
[18]	钱春红, 沈海林, 李振凯. 探讨膝关节囊性病变的磁共振影像(MRI)表现及其诊断价值[J]. 影像研究与医学应用, 2019, 3(18): 54-55.
	QIAN C H, SHEN H L, LI Z K. To investigate the magnetic resonance imaging (MRI) manifestations of cystic lesions of knee joint and its diagnostic value[J]. Journal of Imaging Research and Medical Applications, 2019, 3(18): 54-55 (in Chinese).
[19]	CHEN L C, YANG Y, WANG J, et al. Attention to scale: scale-aware semantic image segmentation[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 3640-3649.
[20]	LI L H, TANG S, ZHANG Y D, et al. GLA: global-local attention for image description[J]. IEEE Transactions on Multimedia, 2018, 20(3): 726-737. DOI URL
[21]	SONG C H, HAN H J, AVRITHIS Y. All the attention You need: global-local, spatial-channel attention for image retrieval[C]// 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2022: 2754-2763.
[22]	KENDALL A, GAL Y, CIPOLLA R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7482-7491.
[23]	MOHAJER B, DOLATSHAHI M, MORADI K, et al. Role of thigh muscle changes in knee osteoarthritis outcomes: osteoarthritis initiative data[J]. Radiology, 2022, 305(1): 169-178. DOI URL
[24]	WANG Z B, LV Y K, WU R L, et al. Review of GrabCut in image processing[J]. Mathematics, 2023, 11(8): 1965. DOI URL

参数	配置
操作系统	Centos Linux release 7.9.2009
CPU	Intel(R) Xeon(R) Gold 6338 CPU @2.00 GHz
GPU	Nvidia Tesla A100 80 G
编程语言	Python 3.8.13
深度学习框架	Tensorflow2.10.0
加速环境	CUDA11.6+cudnn8.4.1

参数	配置
操作系统	Centos Linux release 7.9.2009
CPU	Intel(R) Xeon(R) Gold 6338 CPU @2.00 GHz
GPU	Nvidia Tesla A100 80 G
编程语言	Python 3.8.13
深度学习框架	Tensorflow2.10.0
加速环境	CUDA11.6+cudnn8.4.1

方法		检测任务						分割任务
方法	FPS	AP25	AP50	AP75	AR25	AR50	AR75	AP25	AP50	AP75	AR25	AR50	AR75
YOLOv3	46.702	49.3	75.6	55.9	64.6	63.2	75.0	-	-	-	-	-	-
YOLOv7	62.059	62.6	94.1	71.7	59.2	73.4	89.0	-	-	-	-	-	-
Centernet	117.959	38.2	81.6	28.7	47.5	56.1	71.3	-	-	-	-	-	-
Faster R-CNN	44.786	50.2	75.7	55.2	63.7	63.2	81.7	-	-	-	-	-	-
Cascade Mask R-CNN	3.525	66.4	89.2	71.4	74.3	77.3	75.6	58.2	89.4	66.5	67.3	64.4	67.8
Mask R-CNN	5.222	65.6	88.4	76.5	72.7	78.7	80.0	56.5	89.6	64.2	64	67.2	72.4
Ours	6.286	69.1	92.3	81.6	76.6	81.7	83.0	59.8	92.8	68.8	68	69.3	74.9

方法		检测任务						分割任务
方法	FPS	AP25	AP50	AP75	AR25	AR50	AR75	AP25	AP50	AP75	AR25	AR50	AR75
YOLOv3	46.702	49.3	75.6	55.9	64.6	63.2	75.0	-	-	-	-	-	-
YOLOv7	62.059	62.6	94.1	71.7	59.2	73.4	89.0	-	-	-	-	-	-
Centernet	117.959	38.2	81.6	28.7	47.5	56.1	71.3	-	-	-	-	-	-
Faster R-CNN	44.786	50.2	75.7	55.2	63.7	63.2	81.7	-	-	-	-	-	-
Cascade Mask R-CNN	3.525	66.4	89.2	71.4	74.3	77.3	75.6	58.2	89.4	66.5	67.3	64.4	67.8
Mask R-CNN	5.222	65.6	88.4	76.5	72.7	78.7	80.0	56.5	89.6	64.2	64	67.2	72.4
Ours	6.286	69.1	92.3	81.6	76.6	81.7	83.0	59.8	92.8	68.8	68	69.3	74.9

方法	FPS	检测任务						分割任务
方法	FPS	AP25	AP50	AP75	AR25	AR50	AR75	AP25	AP50	AP75	AR25	AR50	AR75
Ours	6.286	69.1	92.3	81.6	76.6	81.7	83	59.8	92.8	69.8	68	69.3	74.9
Mask R-CNN	5.222	65.6	88.4	76.5	72.7	78.7	80	56.5	89.6	64.2	64	67.2	72.4
Mask R-CNN + GLA	5.885	66.4	90.7	78.9	75.9	78.8	77.8	57.6	88.3	65	64.7	68.5	73.3
Mask R-CNN + MUL	7.884	68.9	91.8	81.6	76.4	81.3	82.9	59.1	92.2	68.8	66.2	69.1	73.2