融合全局-局部注意模块的Mask R-CNN膝关节囊肿检测方法

doi:10.11996/JG.j.2095-302X.2023061183

图学学报 ›› 2023, Vol. 44 ›› Issue (6): 1183-1190.DOI: 10.11996/JG.j.2095-302X.2023061183

• 图像处理与计算机视觉 • 上一篇下一篇

融合全局-局部注意模块的Mask R-CNN膝关节囊肿检测方法

张丽媛¹(), 赵海蓉¹, 何巍¹(), 唐雄风²

1.长春理工大学计算机科学技术学院，吉林长春 130022
2.吉林大学第二医院骨科医疗中心，吉林长春 130041

收稿日期:2023-06-25 接受日期:2023-08-28 出版日期:2023-12-31 发布日期:2023-12-17
通讯作者: 何巍(1978-)，女，教授，博士。主要研究方向为机器学习和医学图像处理。E-mail：hewei@cust.edu.cn
作者简介:
张丽媛(1990-)，女，副教授，博士。主要研究方向为计算机视觉和医学图像处理。E-mail：zhangliyuanzly@cust.edu.cn
基金资助:
国家自然科学基金项目(U21A20390);吉林省自然科学基金项目(YDZJ202101ZYTS036)

Knee cysts detection algorithm based on Mask R-CNN integrating global-local attention module

ZHANG Li-yuan¹(), ZHAO Hai-rong¹, HE Wei¹(), TANG Xiong-feng²

1. School of Computer Science and Technology, Changchun University of Science and Technology, Changchun Jilin 130022, China
2. Orthopedic Medical Center, Jilin University Second Hospital, Changchun Jilin 130041, China

Received:2023-06-25 Accepted:2023-08-28 Online:2023-12-31 Published:2023-12-17
Contact: HE Wei (1978-), professor, Ph.D. Her main research interests cover machine learning and medical image analysis. E-mail：hewei@cust.edu.cn
About author:
ZHANG Li-yuan (1990-), associate professor, Ph.D. Her main research interests cover computer vision and medical image analysis.
E-mail：zhangliyuanzly@cust.edu.cn
Supported by:
National Natural Science Foundation of China(U21A20390);Natural Science Foundation of Jilin Province(YDZJ202101ZYTS036)

摘要/Abstract

摘要：

精准的膝关节囊肿检测是辅助多种膝关节疾病进行早期诊断和治疗的有效手段。然而，由于MR影像中膝关节囊肿与关节内积液和囊肿型肿瘤等其他病灶在影像特征上比较相似，导致检测任务难度较大。因此，提出了一种融合全局-局部注意模块的Mask R-CNN多任务学习模型，同时实现MR影像中膝关节囊肿的自动识别、检测与分割。首先，该方法利用通道注意力机制，将膝关节影像的全局特征和局部特征进行加权融合，形成具有多尺度信息的特征图，为模型提供更准确的判别特征。其次，引入多任务不确定性损失函数，采用同方差不确定性表明每个任务的相对置信度，对任务权重进行自适应调整，可以自动搜索最优解。最后，使用GrabCut方法基于预标记边界框生成掩码，进一步构建膝关节MR影像数据集，提升了数据标注的质量和效率。实验结果表明，该方法可以准确识别膝关节MR影像中的囊肿病灶，同时检测和分割平均精度分别达到了92.3%和92.8%，效果上优于其他对比方法。

关键词: 膝关节囊肿, 多任务学习, 注意力机制, 多尺度特征融合, Mask R-CNN

Abstract:

Accurately detecting knee cysts is an effective means for facilitating early diagnosis and treatment of various knee-related diseases. However, the task of detecting knee cysts can be challenging due to their imaging features' similarity to other lesions in MR imaging, such as intra-articular effusion and cystic tumors. Therefore, a Mask R-CNN multi-task learning model incorporating global-local attention modules was proposed to simultaneously implement the automatic recognition, detection, and segmentation of knee cysts in MRI. Firstly, the method utilized the channel attention mechanism to achieve the weighted fusion of global and local features of knee images, forming a feature map with multi-scale information. This map provided more accurate discriminative features for the model. Secondly, a multi-task uncertainty loss function was introduced, which employed homoskedasticity uncertainty to indicate the relative confidence of each task. It adaptively adjusted the task weights, and automatically searched for the optimal solution. Finally, the GrabCut method was utilized to generate masks based on pre-labeled bounding boxes to further construct knee MRI datasets, enhancing the quality and efficiency of data annotation. The experimental results demonstrated that the proposed method could accurately identify cystic knee lesions in MRI, with an average accuracy of 92.3% for detection and 92.8% for segmentation. These results outperformed other comparison methods in terms of effectiveness.

Key words: knee cysts, multi-task learning, attention mechanism, multi-scale feature fusion, Mask R-CNN

中图分类号:

TP391

张丽媛, 赵海蓉, 何巍, 唐雄风. 融合全局-局部注意模块的Mask R-CNN膝关节囊肿检测方法[J]. 图学学报, 2023, 44(6): 1183-1190.

ZHANG Li-yuan, ZHAO Hai-rong, HE Wei, TANG Xiong-feng. Knee cysts detection algorithm based on Mask R-CNN integrating global-local attention module[J]. Journal of Graphics, 2023, 44(6): 1183-1190.

图/表 10

参考文献 24

[1]	DAINESE P, WYNGAERT K V, DE MITS S, et al. Association between knee inflammation and knee pain in patients with knee osteoarthritis: a systematic review[J]. Osteoarthritis and Cartilage, 2022, 30(4): 516-534. DOI URL
[2]	PERRY T A, O'NEILL T W, TOLSTYKH I, et al. Magnetic resonance imaging-assessed subchondral cysts and incident knee pain and knee osteoarthritis: data from the multicenter osteoarthritis study[J]. Arthritis & Rheumatology, 2022, 74(1): 60-69.
[3]	艾阳平. 100例膝关节囊肿的高频超声与MRI分析[J]. 影像研究与医学应用, 2021, 5(10): 107-108.
	AI Y P. High frequency ultrasound and MRI analysis of 100 cases of knee joint cyst[J]. Journal of Imaging Research and Medical Applications, 2021, 5(10): 107-108 (in Chinese).
[4]	DAINESE P, WYNGAERT K V, DE MITS S, et al. Association between knee inflammation and knee pain in patients with knee osteoarthritis: a systematic review[J]. Osteoarthritis and Cartilage, 2022, 30(4): 516-534. DOI URL
[5]	DING Y, ACOSTA R, ENGUIX V, et al. Using deep convolutional neural networks for neonatal brain image segmentation[J]. Frontiers in Neuroscience, 2020, 14: 1-10. DOI URL
[6]	MEHTA D, GAL Y, BOWLING A, et al. Out-of-distribution detection for long-tailed and fine-grained skin lesion images[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2022: 732-742.
[7]	KHAN S, AZAM B, YAO Y C, et al. Deep collaborative network with alpha matte for precise knee tissue segmentation from MRI[J]. Computer Methods and Programs in Biomedicine, 2022, 222: 106963. DOI URL
[8]	AWAN M J, RAHIM M S M, SALIM N, et al. Efficient detection of knee anterior cruciate ligament from magnetic resonance imaging using deep learning approach[J]. Diagnostics, 2021, 11(1): 105. DOI URL
[9]	HUO X Z, SUN G, TIAN S W, et al. HiFuse: hierarchical multi-scale feature fusion network for medical image classification[EB/OL]. (2022-09-21) [2023-01-18]. https://arxiv.org/abs/2209.10218.pdf.
[10]	LIU S S, WANG H L, LI Y, et al. AHU-MultiNet: adaptive loss balancing based on homoscedastic uncertainty in multi-task medical image segmentation network[J]. Computers in Biology and Medicine, 2022, 150: 106157. DOI URL
[11]	HAQUE A, IMRAN A A Z, WANG A, et al. Generalized multi-task learning from substantially unlabeled multi-source medical image data[EB/OL]. (2021-10-25) [2023-01-18]. https://arxiv.org/abs/2110.13185.pdf.
[12]	TIAN Y N, YANG G D, WANG Z, et al. Apple detection during different growth stages in orchards using the improved YOLO-V3 model[J]. Computers and Electronics in Agriculture, 2019, 157: 417-426. DOI URL
[13]	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 7464-7475.
[14]	ZHOU X Y, WANG D Q, KRÄHENBÜHL P. Objects as points[EB/OL]. (2019-04-16) [2023-01-18]. https://arxiv.org/abs/1904.07850.pdf.
[15]	CHEN Y H, WANG H R, LI W, et al. Scale-aware domain adaptive faster R-CNN[J]. International Journal of Computer Vision, 2021, 129(7): 2223-2243. DOI
[16]	XU B B, WANG W S, FALZON G, et al. Automated cattle counting using Mask R-CNN in quadcopter vision system[J]. Computers and Electronics in Agriculture, 2020, 171: 105300. DOI URL
[17]	CAI Z W, VASCONCELOS N. Cascade R-CNN: high quality object detection and instance segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 43(5): 1483-1498. DOI URL
[18]	钱春红, 沈海林, 李振凯. 探讨膝关节囊性病变的磁共振影像(MRI)表现及其诊断价值[J]. 影像研究与医学应用, 2019, 3(18): 54-55.
	QIAN C H, SHEN H L, LI Z K. To investigate the magnetic resonance imaging (MRI) manifestations of cystic lesions of knee joint and its diagnostic value[J]. Journal of Imaging Research and Medical Applications, 2019, 3(18): 54-55 (in Chinese).
[19]	CHEN L C, YANG Y, WANG J, et al. Attention to scale: scale-aware semantic image segmentation[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 3640-3649.
[20]	LI L H, TANG S, ZHANG Y D, et al. GLA: global-local attention for image description[J]. IEEE Transactions on Multimedia, 2018, 20(3): 726-737. DOI URL
[21]	SONG C H, HAN H J, AVRITHIS Y. All the attention You need: global-local, spatial-channel attention for image retrieval[C]// 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2022: 2754-2763.
[22]	KENDALL A, GAL Y, CIPOLLA R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7482-7491.
[23]	MOHAJER B, DOLATSHAHI M, MORADI K, et al. Role of thigh muscle changes in knee osteoarthritis outcomes: osteoarthritis initiative data[J]. Radiology, 2022, 305(1): 169-178. DOI URL
[24]	WANG Z B, LV Y K, WU R L, et al. Review of GrabCut in image processing[J]. Mathematics, 2023, 11(8): 1965. DOI URL

参数	配置
操作系统	Centos Linux release 7.9.2009
CPU	Intel(R) Xeon(R) Gold 6338 CPU @2.00 GHz
GPU	Nvidia Tesla A100 80 G
编程语言	Python 3.8.13
深度学习框架	Tensorflow2.10.0
加速环境	CUDA11.6+cudnn8.4.1

参数	配置
操作系统	Centos Linux release 7.9.2009
CPU	Intel(R) Xeon(R) Gold 6338 CPU @2.00 GHz
GPU	Nvidia Tesla A100 80 G
编程语言	Python 3.8.13
深度学习框架	Tensorflow2.10.0
加速环境	CUDA11.6+cudnn8.4.1

方法		检测任务						分割任务
方法	FPS	AP25	AP50	AP75	AR25	AR50	AR75	AP25	AP50	AP75	AR25	AR50	AR75
YOLOv3	46.702	49.3	75.6	55.9	64.6	63.2	75.0	-	-	-	-	-	-
YOLOv7	62.059	62.6	94.1	71.7	59.2	73.4	89.0	-	-	-	-	-	-
Centernet	117.959	38.2	81.6	28.7	47.5	56.1	71.3	-	-	-	-	-	-
Faster R-CNN	44.786	50.2	75.7	55.2	63.7	63.2	81.7	-	-	-	-	-	-
Cascade Mask R-CNN	3.525	66.4	89.2	71.4	74.3	77.3	75.6	58.2	89.4	66.5	67.3	64.4	67.8
Mask R-CNN	5.222	65.6	88.4	76.5	72.7	78.7	80.0	56.5	89.6	64.2	64	67.2	72.4
Ours	6.286	69.1	92.3	81.6	76.6	81.7	83.0	59.8	92.8	68.8	68	69.3	74.9

方法		检测任务						分割任务
方法	FPS	AP25	AP50	AP75	AR25	AR50	AR75	AP25	AP50	AP75	AR25	AR50	AR75
YOLOv3	46.702	49.3	75.6	55.9	64.6	63.2	75.0	-	-	-	-	-	-
YOLOv7	62.059	62.6	94.1	71.7	59.2	73.4	89.0	-	-	-	-	-	-
Centernet	117.959	38.2	81.6	28.7	47.5	56.1	71.3	-	-	-	-	-	-
Faster R-CNN	44.786	50.2	75.7	55.2	63.7	63.2	81.7	-	-	-	-	-	-
Cascade Mask R-CNN	3.525	66.4	89.2	71.4	74.3	77.3	75.6	58.2	89.4	66.5	67.3	64.4	67.8
Mask R-CNN	5.222	65.6	88.4	76.5	72.7	78.7	80.0	56.5	89.6	64.2	64	67.2	72.4
Ours	6.286	69.1	92.3	81.6	76.6	81.7	83.0	59.8	92.8	68.8	68	69.3	74.9

方法	FPS	检测任务						分割任务
方法	FPS	AP25	AP50	AP75	AR25	AR50	AR75	AP25	AP50	AP75	AR25	AR50	AR75
Ours	6.286	69.1	92.3	81.6	76.6	81.7	83	59.8	92.8	69.8	68	69.3	74.9
Mask R-CNN	5.222	65.6	88.4	76.5	72.7	78.7	80	56.5	89.6	64.2	64	67.2	72.4
Mask R-CNN + GLA	5.885	66.4	90.7	78.9	75.9	78.8	77.8	57.6	88.3	65	64.7	68.5	73.3
Mask R-CNN + MUL	7.884	68.9	91.8	81.6	76.4	81.3	82.9	59.1	92.2	68.8	66.2	69.1	73.2

融合全局-局部注意模块的Mask R-CNN膝关节囊肿检测方法

Knee cysts detection algorithm based on Mask R-CNN integrating global-local attention module

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 24

相关文章 15

编辑推荐

Metrics

本文评价

[1]	郭宗洋, 刘立东, 蒋东华, 刘子翔, 朱熟康, 陈京华 . 基于语义引导神经网络的人体动作识别算法 [J]. 图学学报, 2024, 45(1): 26-34.
[2]	翟永杰, 赵晓瑜, 王璐瑶, 王亚茹, 宋晓轲, 朱浩硕. IDD-YOLOv7：一种用于输电线路绝缘子多缺陷的轻量化检测方法 [J]. 图学学报, 2024, 45(1): 90-101.
[3]	古天骏, 熊苏雅, 林晓. 基于SASGAN的戏剧脸谱多样化生成 [J]. 图学学报, 2024, 45(1): 102-111.
[4]	崔克彬, 焦静颐 . 基于MCB-FAH-YOLOv8的钢材表面缺陷检测算法 [J]. 图学学报, 2024, 45(1): 112-125.
[5]	魏陈浩, 杨睿, 刘振丙, 蓝如师, 孙希延, 罗笑南. 具有双层路由注意力的YOLOv8道路场景目标检测方法[J]. 图学学报, 2023, 44(6): 1104-1111.
[6]	丁建川, 肖金桐, 赵可新, 贾冬青, 崔炳德, 杨鑫. 基于脉冲神经网络的复杂场景导航避障算法[J]. 图学学报, 2023, 44(6): 1121-1129.
[7]	张驰, 张效娟, 赵洋, 杨帆. 基于调色板的半交互式低照度唐卡图像增强[J]. 图学学报, 2023, 44(6): 1202-1211.
[8]	杨陈成, 董秀成, 侯兵, 张党成, 向贤明, 冯琪茗. 基于参考的Transformer纹理迁移深度图像超分辨率重建[J]. 图学学报, 2023, 44(5): 861-867.
[9]	宋焕生, 文雅, 孙士杰, 宋翔宇, 张朝阳, 李旭. 基于改进教师学生网络的隧道火灾检测[J]. 图学学报, 2023, 44(5): 978-987.
[10]	李利霞, 王鑫, 王军, 张又元. 基于特征融合与注意力机制的无人机图像小目标检测算法[J]. 图学学报, 2023, 44(4): 658-666.
[11]	李鑫, 普园媛, 赵征鹏, 徐丹, 钱文华. 内容语义和风格特征匹配一致的艺术风格迁移[J]. 图学学报, 2023, 44(4): 699-709.
[12]	余伟群, 刘佳涛, 张亚萍. 融合注意力的拉普拉斯金字塔单目深度估计[J]. 图学学报, 2023, 44(4): 728-738.
[13]	胡欣, 周运强, 肖剑, 杨杰. 基于改进YOLOv5的螺纹钢表面缺陷检测[J]. 图学学报, 2023, 44(3): 427-437.
[14]	郝鹏飞, 刘立群, 顾任远. YOLO-RD-Apple果园异源图像遮挡果实检测模型[J]. 图学学报, 2023, 44(3): 456-464.
[15]	罗文宇, 傅明月. 基于YoloX-ECA模型的非法野泳野钓现场监测技术[J]. 图学学报, 2023, 44(3): 465-472.

方法	检测任务						分割任务
方法	AP25	AP50	AP75	AR25	AR50	AR75	AP25	AP50	AP75	AR25	AR50	AR75
多任务不确定性损失	68.9	91.8	81.6	76.4	81.3	82.9	59.1	92.2	68.8	66.2	69.1	73.2
$\lambda _{\text{cls}}^{r}=\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=1$	65.6	88.4	76.5	72.7	78.7	80.0	56.5	89.6	64.2	64.0	67.2	72.4
$\lambda _{\text{cls}}^{r}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=1$	65.1	88.1	77.4	74.3	80.1	81.9	55.3	88.3	63.6	65.5	66.8	71.9
$\lambda _{\text{box}}^{r}=10,\lambda _{\text{cls}}^{r}=\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=1$	65.8	89.7	78.1	74.4	80.6	79.5	55.9	90.3	64	64.9	68.8	70.5
$\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=10,\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=1$	65.3	89.1	75.4	75.3	80.9	79.0	55.1	88.7	62.1	65.8	67.6	67.1
$\lambda _{\text{mask}}^{m}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=1$	67.4	89.8	79.5	76.3	80.1	81.9	57.0	90.0	66	65.4	68.6	72.4
$\lambda _{\text{cls}}^{m}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=\lambda _{\text{mask}}^{m}=\lambda _{\text{box}}^{m}=1$	67.2	89.9	79.1	75.8	79.8	81.1	57.4	90.4	68.5	65.7	68.5	71.0
$\lambda _{\text{box}}^{m}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=\lambda _{\text{mask}}^{m}=\lambda _{\text{cls}}^{m}=1$	64.8	88.4	77.6	74.9	80.3	82.9	55.7	89.8	63.7	65.6	67.8	71.0
$\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=\lambda _{\text{cls}}^{m}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=1$	65.0	88.4	75.3	73.9	80.3	82.9	55.9	89.8	64.8	64.4	67.6	70.0

方法	检测任务						分割任务
方法	AP25	AP50	AP75	AR25	AR50	AR75	AP25	AP50	AP75	AR25	AR50	AR75
多任务不确定性损失	68.9	91.8	81.6	76.4	81.3	82.9	59.1	92.2	68.8	66.2	69.1	73.2
$\lambda _{\text{cls}}^{r}=\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=1$	65.6	88.4	76.5	72.7	78.7	80.0	56.5	89.6	64.2	64.0	67.2	72.4
$\lambda _{\text{cls}}^{r}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=1$	65.1	88.1	77.4	74.3	80.1	81.9	55.3	88.3	63.6	65.5	66.8	71.9
$\lambda _{\text{box}}^{r}=10,\lambda _{\text{cls}}^{r}=\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=1$	65.8	89.7	78.1	74.4	80.6	79.5	55.9	90.3	64	64.9	68.8	70.5
$\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=10,\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=1$	65.3	89.1	75.4	75.3	80.9	79.0	55.1	88.7	62.1	65.8	67.6	67.1
$\lambda _{\text{mask}}^{m}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=\lambda _{\text{cls}}^{m}=\lambda _{\text{box}}^{m}=1$	67.4	89.8	79.5	76.3	80.1	81.9	57.0	90.0	66	65.4	68.6	72.4
$\lambda _{\text{cls}}^{m}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=\lambda _{\text{mask}}^{m}=\lambda _{\text{box}}^{m}=1$	67.2	89.9	79.1	75.8	79.8	81.1	57.4	90.4	68.5	65.7	68.5	71.0
$\lambda _{\text{box}}^{m}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=\lambda _{\text{mask}}^{m}=\lambda _{\text{cls}}^{m}=1$	64.8	88.4	77.6	74.9	80.3	82.9	55.7	89.8	63.7	65.6	67.8	71.0
$\lambda _{\text{box}}^{m}=\lambda _{\text{mask}}^{m}=\lambda _{\text{cls}}^{m}=10,\lambda _{\text{box}}^{r}=\lambda _{\text{cls}}^{r}=1$	65.0	88.4	75.3	73.9	80.3	82.9	55.9	89.8	64.8	64.4	67.6	70.0