Adapter fine-tuning SAM and low-frequency fusion for semantic segmentation of remote sensing images

doi:10.11996/JG.j.2095-302X.2026030543

Abstract

Abstract:

To address the problems of low small-target segmentation accuracy, insufficient cross-modal feature extraction and unsatisfactory fusion effect in multimodal remote sensing image semantic segmentation models, a dual-branch multimodal network Segment Anything Model Low-Frequency Fusion (SAMLoF), based on adapter optimization and low-frequency information extraction and fusion was proposed. By efficiently fine-tuning the parameters of the Segment Anything Model image encoder through the proposed joint adapter structure, the model could learn specific knowledge in the field of remote sensing while maintaining its general capabilities. The specific low-frequency information input modules for different modal images were designed to provide more large-scale environmental structures and contextual information for the model through the Fast Fourier Transform. An efficient feature fusion module based on attention was proposed to promote the full fusion of relevant information in different modal features. SAMLoF was evaluated and compared with 15 methods on the Vaihingen and Potsdam datasets. Both mF1 and mIoU metrics of SAMLoF reached the optimal values. The results showed that SAMLoF could effectively extract and fuse different modal features in remote sensing images, especially generating accurate and smooth boundary contours for complex small target objects.

Key words: semantic segmentation of remote sensing images, segment anything model, adapter fine-tuning low-frequency information, feature fusion

CLC Number:

TP751
TP391

LI Jingtao, FENG Jun, ZHAO Zhihong. Adapter fine-tuning SAM and low-frequency fusion for semantic segmentation of remote sensing images[J]. Journal of Graphics, 2026, 47(3): 543-552.

Figures/Tables 15

References 28

[1]	WANG L B, LI R, DUAN C X, et al. A novel transformer based semantic segmentation scheme for fine-resolution remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 6506105.
[2]	郭丹青, 符颖, 朱烨, 等. 自注意力多尺度特征融合的遥感图像语义分割算法[J]. 计算机辅助设计与图形学学报, 2023, 35(8): 1259-1268.
	GUO D Q, FU Y, ZHU Y, et al. Semantic segmentation of remote sensing image via self-attention-based multi-scale feature fusion[J]. Journal of Computer-Aided Design & Computer Graphics, 2023, 35(8): 1259-1268(in Chinese).
[3]	CHEN J N, MEI J R, LI X H, et al. TransUNet: rethinking the u-net architecture design for medical image segmentation through the lens of transformers[J]. Medical Image Analysis, 2024, 97: 103280. DOI URL
[4]	WANG D, ZHANG J, DU B, et al. SAMRS: scaling-up remote sensing segmentation dataset with segment anything model[C]//The 37th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2023: 385.
[5]	KIRILLOV A, MINTUN E, RAVI N, et al. Segment anything[C]//IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 3992-4003.
[6]	MA X P, WU Q Q, ZHAO X Y, et al. SAM-assisted remote sensing imagery semantic segmentation with object and boundary constraints[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5636916.
[7]	张仕洁, 张斌, 赵文豪. 基于CNN和视觉状态空间的遥感影像语义分割[J]. 计算机应用研究, 2025, 42(5): 1583-1588.
	ZHANG S J, ZHANG B, ZHAO W H. CNN and visual state space based semantic segmentation of remote sensing images[J]. Application Research of Computers, 2025, 42(5): 1583-1588(in Chinese).
[8]	LIU Y, TIAN Y J, ZHAO Y Z, et al. VMamba: visual state space model[C]//The 38th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2024: 3273.
[9]	ZHU E Z, CHEN Z, WANG D K, et al. UNetMamba: an efficient UNet-like mamba for semantic segmentation of high-resolution remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2025, 22: 6001205.
[10]	HE S M, YANG H Q, ZHANG X Y, et al. MFTransNet: a multi-modal fusion with CNN-transformer network for semantic segmentation of HSR remote sensing images[J]. Mathematics, 2023, 11(3): 722. DOI URL
[11]	Ma X P, ZHANG X K, PUN M O, et al. A multilevel multimodal fusion transformer for remote sensing semantic segmentation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5403215.
[12]	李智杰, 程鑫, 李昌华, 等. 跨模态多层特征融合的遥感影像语义分割[J]. 计算机科学与探索, 2025, 19(4): 989-1000. DOI
	LI Z J, CHENG X, LI C H, et al. Cross-modal multi-level feature fusion for semantic segmentation of remote sensing images[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(4): 989-1000(in Chinese).
[13]	胡宇翔, 余长宏, 高明. 多模态融合的遥感图像语义分割网络[J]. 计算机工程与应用, 2024, 60(15): 234-242. DOI
	HU Y X, YU C H, GAO M. Remote sensing image semantic segmentation network based on multimodal fusion[J]. Computer Engineering and Applications, 2024, 60(15): 234-242(in Chinese). DOI
[14]	YAN Z Y, LI J X, LI X X, et al. RingMo-SAM: a foundation model for segment anything in multimodal remote-sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5625716.
[15]	ZHANG J L, ZHOU Z L, MAI G C, et al. Text2Seg: zero-shot remote sensing image semantic segmentation via text-guided visual foundation models[C]//The 7th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. New York: ACM, 2024: 63-66.
[16]	ZHAO J Q, ZHANG M, ZHOU Z H, et al. CFFormer: a cross-fusion transformer framework for the semantic segmentation of multisource remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 4401117.
[17]	WANG L B, LI R, ZHANG C, et al. UNetFormer: a UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 190: 196-214. DOI URL
[18]	HOULSBY N, GIURGIU A, JASTRZEBSKI S, et al. Parameter- efficient transfer learning for NLP[EB/OL]. [2025-12-18]. https://proceedings.mlr.press/v97/houlsby19a.html.
[19]	MA X P, ZHANG X K, PUN M O, et al. MANet: fine-tuning segment anything model for multimodal remote sensing semantic segmentation[EB/OL]. (2025-12-16) [2025-12-18]. https://arxiv.org/pdf/2410.11160.
[20]	MA X P, ZHANG X K, PUN M O. A crossmodal multiscale fusion network for semantic segmentation of remote sensing data[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15: 3463-3474. DOI URL
[21]	LI R, ZHENG S Y, ZHANG C, et al. ABCNet: attentive bilateral contextual network for efficient semantic segmentation of fine-resolution remotely sensed imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 181: 84-98. DOI URL
[22]	LI R, ZHENG S Y, DUAN C X, et al. Multistage attention ResU-Net for semantic segmentation of fine-resolution remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 8009205.
[23]	ZHAO H S, SHI J P, QI X J, et al. Pyramid scene parsing network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 6230-6239.
[24]	MA X P, ZHANG X K, PUN M O. RS³Mamba: visual state space model for remote sensing image semantic segmentation[J]. IEEE Geoscience and Remote Sensing Letters, 2024, 21: 6011405.
[25]	HAZIRBAS C, MA L N, DOMOKOS C, et al. FuseNet: incorporating depth into semantic segmentation via fusion- based CNN architecture[C]//The 13th Asian Conference on Computer Vision on Computer Vision-ACCV 2016. Cham: Springer, 2016: 213-228.
[26]	AUDEBERT N, LE SAUX B, LEFÈVRE S. Beyond RGB: very high resolution urban remote sensing with multimodal deep networks[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 140: 20-32. DOI URL
[27]	SEICHTER D, KÖHLER M, LEWANDOWSKI B, et al. Efficient RGB-D semantic segmentation for indoor scene analysis[C]//2021 IEEE International Conference on Robotics and Automation. New York: IEEE Press, 2021: 13525-13531.
[28]	HOSSEINPOUR H, SAMADZADEGAN F, JAVAN F D. CMGFNet: a deep cross-modal gated fusion network for building extraction from very high-resolution remote sensing images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 184: 96-115. DOI URL

Method	Backbone	OA						mF1	mIoU
Method	Backbone	Bui.	Tre.	Low.	Car	Imp.	Total	mF1	mIoU
ABCNet	ResNet18	94.10	90.81	78.53	64.12	89.70	89.25	85.34	75.20
MAResU-Net	ResNet18	94.84	89.99	79.09	85.89	92.19	90.17	88.54	79.89
PSPNet	ResNet101	94.52	90.17	78.84	79.22	92.03	89.94	86.55	76.96
UNetFormer	ResNet18	96.23	91.85	79.95	86.99	91.85	91.17	89.48	81.97
SAMRS	ResNet18	97.68	92.25	79.42	88.20	92.31	91.73	90.40	82.88
RS3Mamba	R18-Mamba-T	97.40	92.14	79.56	88.15	92.19	91.64	90.34	82.78
UNetMamba	ResT-Lite	98.05	92.34	80.93	88.19	93.15	92.24	90.62	83.28
FuseNet	VGG16	96.28	90.28	78.98	81.37	91.66	90.51	87.71	78.71
VFuseNet	VGG16	95.92	91.36	77.64	76.06	91.85	90.49	87.89	78.92
ESANet	ResNet34	95.69	90.50	77.16	85.46	91.39	90.61	88.18	79.42
CMGFNet	ResNet34	97.75	91.60	80.03	87.28	92.35	91.72	90.00	82.26
CMFNet	VGG16	97.17	90.82	80.37	85.47	92.36	91.40	89.48	81.44
MFTransNet	ResNet34	96.41	91.48	80.09	86.52	92.11	91.22	89.62	81.61
TransUNet	R50-ViT-B	96.48	92.77	76.14	69.56	91.66	90.96	87.34	78.26
FTransUNet	R50-ViT-B	98.20	91.94	81.49	91.27	93.01	92.40	91.21	84.23
SAMLoF	ViT-B	98.35	92.72	81.06	88.32	93.37	92.55	90.80	83.54
SAMLoF	ViT-L	98.77	93.34	81.24	89.65	93.47	92.96	91.69	84.92

Method	Backbone	OA						mF1	mIoU
Method	Backbone	Bui.	Tre.	Low.	Car	Imp.	Total	mF1	mIoU
ABCNet	ResNet18	94.10	90.81	78.53	64.12	89.70	89.25	85.34	75.20
MAResU-Net	ResNet18	94.84	89.99	79.09	85.89	92.19	90.17	88.54	79.89
PSPNet	ResNet101	94.52	90.17	78.84	79.22	92.03	89.94	86.55	76.96
UNetFormer	ResNet18	96.23	91.85	79.95	86.99	91.85	91.17	89.48	81.97
SAMRS	ResNet18	97.68	92.25	79.42	88.20	92.31	91.73	90.40	82.88
RS3Mamba	R18-Mamba-T	97.40	92.14	79.56	88.15	92.19	91.64	90.34	82.78
UNetMamba	ResT-Lite	98.05	92.34	80.93	88.19	93.15	92.24	90.62	83.28
FuseNet	VGG16	96.28	90.28	78.98	81.37	91.66	90.51	87.71	78.71
VFuseNet	VGG16	95.92	91.36	77.64	76.06	91.85	90.49	87.89	78.92
ESANet	ResNet34	95.69	90.50	77.16	85.46	91.39	90.61	88.18	79.42
CMGFNet	ResNet34	97.75	91.60	80.03	87.28	92.35	91.72	90.00	82.26
CMFNet	VGG16	97.17	90.82	80.37	85.47	92.36	91.40	89.48	81.44
MFTransNet	ResNet34	96.41	91.48	80.09	86.52	92.11	91.22	89.62	81.61
TransUNet	R50-ViT-B	96.48	92.77	76.14	69.56	91.66	90.96	87.34	78.26
FTransUNet	R50-ViT-B	98.20	91.94	81.49	91.27	93.01	92.40	91.21	84.23
SAMLoF	ViT-B	98.35	92.72	81.06	88.32	93.37	92.55	90.80	83.54
SAMLoF	ViT-L	98.77	93.34	81.24	89.65	93.47	92.96	91.69	84.92

Method	Backbone	OA						mF1	mIoU
Method	Backbone	Bui.	Tre.	Low.	Car	Imp.	Total	mF1	mIoU
ABCNet	ResNet18	96.23	78.92	86.40	92.92	88.90	87.52	88.14	79.26
MAResU-Net	ResNet18	96.82	83.97	87.70	95.88	92.19	89.82	90.86	83.61
PSPNet	ResNet101	97.03	83.13	85.67	88.81	90.91	88.67	88.92	80.36
UNetFormer	ResNet18	97.69	86.47	87.93	95.91	92.27	90.65	91.71	85.05
SAMRS	ResNet18	97.57	86.62	88.03	96.12	92.35	90.98	91.83	85.37
RS3Mamba	R18-Mamba-T	97.70	86.11	89.53	96.23	91.36	90.49	91.69	85.01
UNetMamba	ResT-Lite	97.98	87.34	87.86	96.05	92.43	91.12	92.05	85.57
FuseNet	VGG16	97.48	85.14	87.31	96.10	92.64	90.58	91.60	84.86
VFuseNet	VGG16	97.23	84.29	89.03	95.49	91.62	90.22	91.26	84.26
ESANet	ResNet34	97.10	85.31	87.81	94.08	92.76	89.74	91.22	84.15
CMGFNet	ResNet34	97.41	86.80	86.68	95.68	92.60	90.21	91.40	84.53
CMFNet	VGG16	97.63	87.40	88.00	95.68	92.84	91.16	92.10	85.63
MFTransNet	ResNet34	97.37	85.71	86.92	96.05	92.45	89.96	91.11	84.04
TransUNet	R50-ViT-B	96.63	82.65	89.98	93.17	91.93	90.01	90.97	83.74
FTransUNet	R50-ViT-B	97.78	88.27	88.48	96.31	93.17	91.34	92.41	86.20
SAMLoF	ViT-B	98.15	87.66	87.94	96.18	92.85	91.35	92.33	85.81
SAMLoF	ViT-L	98.33	88.82	88.38	96.37	93.75	91.66	92.68	86.55

Method	Backbone	OA						mF1	mIoU
Method	Backbone	Bui.	Tre.	Low.	Car	Imp.	Total	mF1	mIoU
ABCNet	ResNet18	96.23	78.92	86.40	92.92	88.90	87.52	88.14	79.26
MAResU-Net	ResNet18	96.82	83.97	87.70	95.88	92.19	89.82	90.86	83.61
PSPNet	ResNet101	97.03	83.13	85.67	88.81	90.91	88.67	88.92	80.36
UNetFormer	ResNet18	97.69	86.47	87.93	95.91	92.27	90.65	91.71	85.05
SAMRS	ResNet18	97.57	86.62	88.03	96.12	92.35	90.98	91.83	85.37
RS3Mamba	R18-Mamba-T	97.70	86.11	89.53	96.23	91.36	90.49	91.69	85.01
UNetMamba	ResT-Lite	97.98	87.34	87.86	96.05	92.43	91.12	92.05	85.57
FuseNet	VGG16	97.48	85.14	87.31	96.10	92.64	90.58	91.60	84.86
VFuseNet	VGG16	97.23	84.29	89.03	95.49	91.62	90.22	91.26	84.26
ESANet	ResNet34	97.10	85.31	87.81	94.08	92.76	89.74	91.22	84.15
CMGFNet	ResNet34	97.41	86.80	86.68	95.68	92.60	90.21	91.40	84.53
CMFNet	VGG16	97.63	87.40	88.00	95.68	92.84	91.16	92.10	85.63
MFTransNet	ResNet34	97.37	85.71	86.92	96.05	92.45	89.96	91.11	84.04
TransUNet	R50-ViT-B	96.63	82.65	89.98	93.17	91.93	90.01	90.97	83.74
FTransUNet	R50-ViT-B	97.78	88.27	88.48	96.31	93.17	91.34	92.41	86.20
SAMLoF	ViT-B	98.15	87.66	87.94	96.18	92.85	91.35	92.33	85.81
SAMLoF	ViT-L	98.33	88.82	88.38	96.37	93.75	91.66	92.68	86.55

EA	LFIM	EFM	Total	mF1	mIoU
			92.12	90.29	82.71
√			92.38	90.50	83.07
	√		92.25	90.46	82.98
		√	92.29	90.55	83.09
√	√		92.42	90.68	83.34
√	√	√	92.55	90.80	83.54