Deep multimodal medical image fusion network based on high-low frequency feature decomposition

doi:10.11996/JG.j.2095-302X.2024010065

Abstract

Abstract:

Multimodal medical image fusion aims to enhance the interpretability and applicability of medical images in clinical settings by leveraging correlations and complementary information across different imaging modalities. However, existing manually designed models often fail to effectively extract critical target features, resulting in issues such as blurred fusion images and loss of textural details. To address this, a novel deep multimodal medical image fusion network based on high-low frequency feature decomposition was proposed. This approach incorporated channel attention and spatial attention mechanisms into the fusion process, allowing for a more intricate fusion of high-low frequency features while preserving both global structure and local textural details. Firstly, the high-frequency features of two modal images were extracted using the pre-trained model VGG-19, and their low-frequency features were extracted through downsampling to form intermediate features between high and low frequencies. Secondly, a residual attention network was embedded in the feature fusion module to sequentially infer attention maps from independent channels and spatial dimensions. These maps were then employed to guide the adaptive feature optimization of input feature maps. Finally, the reconstruction module fused high-low frequency features and output the fusion image. Experimental results on both the Harvard open dataset and a self-created abdominal dataset demonstrated that compared to the source image, the fusion image produced by the proposed method achieved an 8.29% improvement in peak signal-to-noise ratio, 85.07% in structural similarity, 65.67% in correlation coefficient, 46.76% in feature mutual information, and 80.89% in visual fidelity.

Key words: multi-modal medical image fusion, pre-trained model, deep learning, high-low frequency feature extraction, residual attention network

CLC Number:

TP391

WANG Xinyu, LIU Hui, ZHU Jicheng, SHENG Yurui, ZHANG Caiming. Deep multimodal medical image fusion network based on high-low frequency feature decomposition[J]. Journal of Graphics, 2024, 45(1): 65-77.

Figures/Tables 14

References 33

[1]	GAO Y, MA S W, LIU J J, et al. Fusion of medical images based on salient features extraction by PSO optimized fuzzy logic in NSST domain[J]. Biomedical Signal Processing and Control, 2021, 69: 102852. DOI URL
[2]	朱积成, 刘慧, 李珊珊, 等. 结合局部熵与梯度能量的双通道医学图像融合[J/OL]. 计算机辅助设计与图形学学报. [2023-06-18]. https://www.jcad.cn/cn/search.
	ZHU J C, LIU H, LI S S, et al. Two-channel Medical Image Fusion Combining Local Entropy and Gradient Energy[J/OL]. Journal of Computer-Aided Design & Computer Graphics. [2023-06-18]. https://www.jcad.cn/cn/search. (in Chinese).
[3]	XU Z Q J, ZHANG Y Y, XIAO Y Y. Training behavior of deep neural network in frequency domain[C]// International Conference on Neural Information Processing. Cham: Springer, 2019: 264-274.
[4]	WANG K P, ZHENG M Y, WEI H Y, et al. Multi-modality medical image fusion using convolutional neural network and contrast pyramid[J]. Sensors, 2020, 20(8): 2169. DOI URL
[5]	HERMESSI H, MOURALI O, ZAGROUBA E. Convolutional neural network-based multimodal image fusion via similarity learning in the shearlet domain[J]. Neural Computing and Applications, 2018, 30(7): 2029-2045. DOI
[6]	JOSE J, GAUTAM N, TIWARI M, et al. An image quality enhancement scheme employing adolescent identity search algorithm in the NSST domain for multimodal medical image fusion[J]. Biomedical Signal Processing and Control, 2021, 66: 102480. DOI URL
[7]	ZHAO C, YANG P, ZHOU F, et al. MHW-GAN: multidiscriminator hierarchical wavelet generative adversarial network for multimodal image fusion[J/OL]. IEEE Transactions on Neural Networks and Learning Systems. [2023-06-19]. https://ieeexplore.ieee.org/document/10177917.
[8]	LIU Y, CHEN X, PENG H, et al. Multi-focus image fusion with a deep convolutional neural network[J]. Information Fusion, 2017, 36: 191-207. DOI URL
[9]	DENG X, DRAGOTTI P L. Deep convolutional neural network for multi-modal image restoration and fusion[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(10): 3333-3348. DOI URL
[10]	LI S S, ZOU Y H, WANG G J, et al. Infrared and visible image fusion method based on principal component analysis network and multi-scale morphological gradient[J]. Infrared Physics & Technology, 2023, 133: 104810.
[11]	XU H, MA J Y. EMFusion: an unsupervised enhanced medical image fusion network[J]. Information Fusion, 2021, 76: 177-186. DOI URL
[12]	MA J Y, YU W, LIANG P W, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48(C): 11-26.
[13]	ZHANG Y, LIU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118. DOI URL
[14]	LAHOUD F, SÜSSTRUNK S. Zero-learning fast medical image fusion[C]// 2019 22th International Conference on Information Fusion. New York: IEEE Press, 2020: 1-8.
[15]	LIU H, LI S S, ZHU J C, et al. DDIFN: a dual-discriminator multi-modal medical image fusion network[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 19(4): 145:1-145:17.
[16]	ZOPH B, GHIASI G, LIN T Y, et al. Rethinking pre-training and self-training[C]// The 34th International Conference on Neural Information Processing Systems. New York: ACM, 2020: 3833-3845.
[17]	杜鹏飞, 李小勇, 高雅丽. 多模态视觉语言表征学习研究综述[J]. 软件学报, 2021, 32(2): 327-348.
	DU P F, LI X Y, GAO Y L. Survey on multimodal visual language representation learning[J]. Journal of Software, 2021, 32(2): 327-348 (in Chinese).
[18]	KORNBLITH S, SHLENS J, LE Q V. Do better ImageNet models transfer better?[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 2656-2666.
[19]	HENDRYCKS D, LEE K, MAZEIKA M. Using pre-training can improve model robustness and uncertainty[EB/OL]. [2023-06-18]. https://arxiv.org/abs/1901.09960.pdf.
[20]	刘慧, 李珊珊, 高珊珊, 等. 预训练模型特征提取的双对抗磁共振图像融合网络研究[J]. 软件学报, 2023, 34(5): 2134-2151.
	LIU H, LI S S, GAO S S, et al. Research on dual-adversarial MR image fusion network using pre-trained model for feature extraction[J]. Journal of Software, 2023, 34(5): 2134-2151 (in Chinese).
[21]	王军敏, 樊养余, 李祖贺. 基于深度卷积神经网络和迁移学习的纹理图像识别[J]. 计算机辅助设计与图形学学报, 2022, 34(5): 701-710.
	WANG J M, FAN Y Y, LI Z H. Texture image recognition based on deep convolutional neural network and transfer learning[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(5): 701-710 (in Chinese).
[22]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 770-778.
[23]	MNIH V, HEESS N, GRAVES A, et al. Recurrent models of visual attention[EB/OL]. [2023-06-18]. https://arxiv.org/abs/1406.6247.pdf.
[24]	WANG F, JIANG M Q, QIAN C, et al. Residual attention network for image classification[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 6450-6458.
[25]	金燕, 薛智中, 姜智伟. 基于循环残差卷积神经网络的医学图像分割算法[J]. 计算机辅助设计与图形学学报, 2022, 34(8): 1205-1215.
	JIN Y, XUE Z Z, JIANG Z W. Medical image segmentation based on recurrent residual convolution neural network[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(8): 1205-1215 (in Chinese).
[26]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]// European Conference on Computer Vision. Cham: Springer, 2018: 3-19.
[27]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90. DOI URL
[28]	JAGALINGAM P, HEGDE A V. A review of quality metrics for fused image[J]. Aquatic Procedia, 2015, 4: 133-142. DOI URL
[29]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2004, 13(4): 600-612. DOI URL
[30]	QU G H, ZHANG D L, YAN P F. Information measure for performance of image fusion[J]. Electronics Letters, 2002, 38(7): 313-315. DOI URL
[31]	ASUERO A, SAYAGO A, GONZÁLEZ A G. The correlation coefficient: an overview[J]. Critical Reviews in Analytical Chemistry, 2006, 36: 41-59. DOI URL
[32]	HAN Y, CAI Y Z, CAO Y, et al. A new image fusion performance metric based on visual information fidelity[J]. Information Fusion, 2013, 14(2): 127-135. DOI URL
[33]	TAN W, THITØN W, XIANG P, et al. Multi-modal brain image fusion based on multi-level edge-preserving filtering[J]. Biomedical Signal Processing and Control, 2021, 64: 102280. DOI URL

实验方式	PSNR	SSIM	CC	MI	VIF
单个注意力	64.450 2	0.570 1	0.872 8	3.018 9	0.516 8
注意力串联	64.889 4	0.715 8	0.888 1	3.588 7	0.615 6

实验方式	PSNR	SSIM	CC	MI	VIF
单个注意力	64.450 2	0.570 1	0.872 8	3.018 9	0.516 8
注意力串联	64.889 4	0.715 8	0.888 1	3.588 7	0.615 6

实验方式	PSNR	SSIM	CC	MI	VIF
单个注意力	63.103 8	0.615 2	0.847 9	2.240 1	0.444 3
注意力串联	63.250 8	0.692 9	0.844 6	3.596 3	0.725 1

实验方式	PSNR	SSIM	CC	MI	VIF
单个注意力	63.103 8	0.615 2	0.847 9	2.240 1	0.444 3
注意力串联	63.250 8	0.692 9	0.844 6	3.596 3	0.725 1

数据集类别	图像数量/幅	图像分辨率/bit	层间距/mm	病灶数据
CT	329	256×256	5	钙化灶：最大截面：2.3×1.9 cm² 转移瘤直径：4.5 cm
PET	329	256×256	1	肿大淋巴结直：1.4 cm
MRT1/T2	74	256×256	5	肾囊肿：数量3，最大径：0.6 cm，最小径：0.3 cm，形状：类圆形，边界清晰