MSFAFuse: sar and optical image fusion model based on multi-scale feature information and attention mechanism

doi:10.11996/JG.j.2095-302X.2025020300

Abstract

Abstract:

In response to the issue that remote sensing images obtained from a single imaging principle cannot provide rich information, heterogeneous remote sensing image fusion technology has emerged. The imaging of synthetic aperture radar images is not affected by factors such as clouds and weather, but lacks visual observation capability. The imaging of optical images is susceptible to harsh environments, but offer direct viewing effects and target interpretation capabilities. Integrating the two can fully leverage their respective advantages to obtain high-quality images that contain more feature information and possess visual observation capabilities. To fully utilize the different scale features of heterogeneous images, a SAR and optical image fusion model based on multi-scale feature information and attention mechanism (MSFAFuse) was proposed. Firstly, robust feature downsampling was introduced to form the feature extraction part, obtaining multi-scale features corresponding to heterogeneous images. Secondly, a feature enhancement module was employed to enhance the structural features and salient regional features in heterogeneous features at different scales. Then, using a dual-branch fusion module guided by feature information and L1-Norm, was used to fuse the obtained heterogeneous multi-scale features pairwise according to scale. Finally, the fusion results of different scales were input into the image reconstruction module for image reconstruction, resulting in the final fused image. Experiments have shown that the MSFAFuse model can smoothly enhance prominent features while preserving more details and structural information. Compared with existing fusion methods, the model has shown better performance across 10 different indicators. This demonstrated that the MSFAFuse model can effectively fuse optical images and SAR images, providing new insights for the development of their fusion and contributing to the advancement of future remote sensing image fusion technologies.

Key words: image fusion, multi-scale feature, Efficient additive attention, remote sense, deep learning

CLC Number:

PAN Shuyan, LIU Liqun. MSFAFuse: sar and optical image fusion model based on multi-scale feature information and attention mechanism[J]. Journal of Graphics, 2025, 46(2): 300-311.

Figures/Tables 16

References 28

[1]	SHAKYA A, BISWAS M, PAL M. Fusion and classification of SAR and optical data using multi-image color components with differential gradients[J]. Remote Sensing, 2023, 15(1): 274.
[2]	盛佳佳, 杨学志, 董张玉, 等. 基于NSST-IHS变换稀疏表示的SAR与可见光图像融合[J]. 图学学报, 2018, 39(2): 201-208. DOI
	SHENG J J, YANG X Z, DONG Z Y, et al. Fusion of SAR and visible images based on NSST-IHS and sparse representation[J]. Journal of Graphics, 2018, 39(2): 201-208 (in Chinese). DOI
[3]	纪峰, 李泽仁, 常霞, 等. 基于PCA和NSCT变换的遥感图像融合方法[J]. 图学学报, 2017, 38(2): 247-252. DOI
	JI F, LI Z R, CHANG X, et al. Remote sensing image fusion method based on PCA and NSCT transform[J]. Journal of Graphics, 2017, 38(2): 247-252 (in Chinese). DOI
[4]	ZHANG H, SHEN H F, YUAN Q Q, et al. Multispectral and SAR image fusion based on Laplacian pyramid and sparse representation[J]. Remote Sensing, 2022, 14(4): 870.
[5]	LI W S, XIAO X Y, XIAO P H, et al. Change detection in multitemporal SAR images based on slow feature analysis combined with improving image fusion strategy[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15: 3008-3023.
[6]	YE Y X, ZHANG J C, ZHOU L, et al. Optical and SAR image fusion based on complementary feature decomposition and visual saliency features[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5205315.
[7]	HUANG D S, TANG Y L, WANG Q S. An image fusion method of SAR and multispectral images based on non-subsampled shearlet transform and activity measure[J]. Sensors, 2022, 22(18): 7055.
[8]	LI X C, JING D, LI Y C, et al. Multi-band and polarization SAR images colorization fusion[J]. Remote Sensing, 2022, 14(16): 4022.
[9]	LUO J H, ZHOU F, YANG J, et al. DAFCNN: a dual-channel feature extraction and attention feature fusion convolution neural network for SAR image and MS image fusion[J]. Remote Sensing, 2023, 15(12): 3091.
[10]	CHU B C, CHEN J Y, CHEN J, et al. SDCAFNet: a deep convolutional neural network for land-cover semantic segmentation with the fusion of PolSAR and optical images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15: 8928-8942.
[11]	KANG W C, XIANG Y M, WANG F, et al. CFNet: a cross fusion network for joint land cover classification using optical and SAR images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15: 1562-1574.
[12]	WANG Y Y, ZHANG W G, CHEN W D, et al. MFFnet: multimodal feature fusion network for synthetic aperture radar and optical image land cover classification[J]. Remote Sensing, 2024, 16(13): 2459.
[13]	XIA Y, HE W, HUANG Q, et al. SOSSF: landsat-8 image synthesis on the blending of Sentinel-1 and MODIS data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5401619.
[14]	SUN Y C, YAN K J, LI W Z. CycleGAN-based SAR-optical image fusion for target recognition[J]. Remote Sensing, 2023, 15(23): 5569.
[15]	SONG B Z, LIU P, LI J, et al. MLFF-GAN: a multilevel feature fusion with GAN for spatiotemporal remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 4410816.
[16]	SU L J, SUI Y X, YUAN Y. An unmixing-based multi-attention GAN for unsupervised hyperspectral and multispectral image fusion[J]. Remote Sensing, 2023, 15(4): 936.
[17]	WEI J, ZOU H X, SUN L, et al. CFRWD-GAN for SAR-to-optical image translation[J]. Remote Sensing, 2023, 15(10): 2547.
[18]	LU W, CHEN S B, TANG J, et al. A robust feature downsampling module for remote-sensing visual tasks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 4404312.
[19]	SHAKER A, MAAZ M, RASHEED H, et al. SwiftFormer: efficient additive attention for transformer-based real-time mobile vision applications[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 17379-17390.
[20]	YANG D P, PENG B, AL-HUDA Z, et al. An overview of edge and object contour detection[J]. Neurocomputing, 2022, 488: 470-493.
[21]	LI H, WU X J, KITTLER J. RFN-Nest: an end-to-end residual fusion network for infrared and visible images[J]. Information Fusion, 2021, 73: 72-86.
[22]	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]// The 13th European Conference on Computer Vision. Cham: Springer, 2014: 740-755.
[23]	XIANG Y M, TAO R S, WANG F, et al. Automatic registration of optical and SAR images via improved phase congruency model[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 5847-5861.
[24]	HUANG M Y, XU Y, QIAN L X, et al. The QXS-SAROPT dataset for deep learning in SAR-optical data fusion[EB/OL]. [2024-05-29]. https://arxiv.org/abs/2103.08259.
[25]	LI H, WU X J, DURRANI T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[26]	LI H, WU X J. CrossFuse: a novel cross attention mechanism based infrared and visible image fusion approach[J]. Information Fusion, 2024, 103: 102147.
[27]	XU H, MA J Y, JIANG J J, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518.
[28]	WANG Z S, SHAO W Y, CHEN Y L, et al. A cross-scale iterative attentional adversarial fusion network for infrared and visible images[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(8): 3677-3688.

数据集	SAR来源	Opt来源	分辨率/m	大小/像素	图片总数/对	训练图象数量/对	测试图像数量/对
OS dataset^[23]	聚束模式的中国高分 3号C波段传感器	谷歌地球平台	1	256×256	10 692	10 640	52
QXS-SAROPT^[24]	高分3号SAR影像	谷歌地球多光谱影像	1	256×256	20 000	19 984	16

数据集	SAR来源	Opt来源	分辨率/m	大小/像素	图片总数/对	训练图象数量/对	测试图像数量/对
OS dataset^[23]	聚束模式的中国高分 3号C波段传感器	谷歌地球平台	1	256×256	10 692	10 640	52
QXS-SAROPT^[24]	高分3号SAR影像	谷歌地球多光谱影像	1	256×256	20 000	19 984	16

数据集	指标	RFN-Nest	Nestfuse	CrossFuse 1	U2Fusion	CrossFuse 2	MSFAFuse (Ours)
OS dataset	SD	31.313	31.664	32.611	35.250	29.349	40.077
	EN	6.914	6.865	6.931	7.029	6.723	7.141
	MI	1.684	2.282	1.737	1.754	2.473	5.944
	AG	9.583	11.170	12.587	11.716	5.154	8.424
	VIF	0.338	0.401	0.352	0.340	0.497	1.042
	SSIM	0.495	0.467	0.443	0.504	0.504	0.590
	MS-SSIM	0.629	0.600	0.561	0.632	0.609	0.606
	FMI_pixel	0.791	0.817	0.810	0.797	0.796	0.849
	FMI_dct	0.408	0.454	0.475	0.388	0.274	0.494
	FMI_w	0.406	0.468	0.475	0.420	0.307	0.505
QXS-SAROPT	SD	29.054	30.474	33.768	33.454	33.073	42.750
	EN	6.500	6.537	6.779	6.647	6.663	6.907
	MI	2.066	2.355	2.402	2.142	2.250	6.656
	AG	8.444	8.337	9.701	8.957	10.867	10.335
	VIF	0.375	0.410	0.418	0.358	0.424	1.025
	SSIM	0.480	0.501	0.513	0.491	0.479	0.589
	MS-SSIM	0.562	0.572	0.555	0.582	0.537	0.598
	FMI_pixel	0.814	0.819	0.821	0.808	0.801	0.866
	FMI_dct	0.388	0.411	0.402	0.355	0.331	0.535
	FMI_w	0.406	0.429	0.415	0.391	0.341	0.538

数据集	指标	RFN-Nest	Nestfuse	CrossFuse 1	U2Fusion	CrossFuse 2	MSFAFuse (Ours)
OS dataset	SD	31.313	31.664	32.611	35.250	29.349	40.077
	EN	6.914	6.865	6.931	7.029	6.723	7.141
	MI	1.684	2.282	1.737	1.754	2.473	5.944
	AG	9.583	11.170	12.587	11.716	5.154	8.424
	VIF	0.338	0.401	0.352	0.340	0.497	1.042
	SSIM	0.495	0.467	0.443	0.504	0.504	0.590
	MS-SSIM	0.629	0.600	0.561	0.632	0.609	0.606
	FMI_pixel	0.791	0.817	0.810	0.797	0.796	0.849
	FMI_dct	0.408	0.454	0.475	0.388	0.274	0.494
	FMI_w	0.406	0.468	0.475	0.420	0.307	0.505
QXS-SAROPT	SD	29.054	30.474	33.768	33.454	33.073	42.750
	EN	6.500	6.537	6.779	6.647	6.663	6.907
	MI	2.066	2.355	2.402	2.142	2.250	6.656
	AG	8.444	8.337	9.701	8.957	10.867	10.335
	VIF	0.375	0.410	0.418	0.358	0.424	1.025
	SSIM	0.480	0.501	0.513	0.491	0.479	0.589
	MS-SSIM	0.562	0.572	0.555	0.582	0.537	0.598
	FMI_pixel	0.814	0.819	0.821	0.808	0.801	0.866
	FMI_dct	0.388	0.411	0.402	0.355	0.331	0.535
	FMI_w	0.406	0.429	0.415	0.391	0.341	0.538

数据集	SD	EN	MI	AG	VIF	SSIM	MS-SSIM	下采样
OS dataset	30.690	6.789	2.522	9.070	0.595	0.545	0.580	Maxpooling
OS dataset	40.077	7.141	5.944	8.424	1.042	0.590	0.606	RFD
QXS-SAROPT	31.731	6.588	2.706	11.041	0.548	0.446	0.495	Maxpooling
QXS-SAROPT	42.750	6.907	6.656	10.335	1.025	0.589	0.598	RFD