适配器微调SAM与低频融合的遥感图像语义分割

doi:10.11996/JG.j.2095-302X.2026030543

图学学报 ›› 2026, Vol. 47 ›› Issue (3): 543-552.DOI: 10.11996/JG.j.2095-302X.2026030543

• 图像处理与计算机视觉 • 上一篇下一篇

适配器微调SAM与低频融合的遥感图像语义分割

李景涛¹^,², 封筠¹^,²(), 赵志宏¹^,²

¹ 石家庄铁道大学信息科学与技术学院，河北石家庄 050043
² 河北省电磁环境效应与信息处理重点实验室，河北石家庄 050043

收稿日期:2025-10-16 接受日期:2026-02-06 出版日期:2026-06-30 发布日期:2026-06-30
通讯作者:封筠，E-mail：fengjun@stdu.edu.cn
基金资助:
河北省自然科学基金(F2024210005);石家庄市市级科技计划资助(2512100201A);中国铁路南宁局集团有限公司科研计划项目(工25-2)

Adapter fine-tuning SAM and low-frequency fusion for semantic segmentation of remote sensing images

LI Jingtao¹^,², FENG Jun¹^,²(), ZHAO Zhihong¹^,²

¹ School of Information Science and Technology, Shijiazhuang Tiedao University, Shijiazhuang Hebei 050043, China
² Hebei Key Laboratory of Electromagnetic Environmental Effects and Information Processing, Shijiazhuang Hebei 050043, China

Received:2025-10-16 Accepted:2026-02-06 Published:2026-06-30 Online:2026-06-30
Contact: FENG Jun，E-mail：fengjun@stdu.edu.cn
Supported by:
Natural Science Foundation of Hebei Province(F2024210005);Science and Technology Program of Shijiazhuang(2512100201A);Scientific Research Plan Project of China Railway Nanning Bureau Group Co., Ltd.(G25-2)

摘要/Abstract

摘要：

针对现有多模态遥感图像语义分割模型具有小目标分割精度低，跨模态特征提取不充分及融合效果不理想的问题，提出一种基于适配器优化和低频信息提取与融合的双分支多模态网络SAMLoF。通过联合适配器结构，高效微调分割一切模型的图像编码器参数，以使模型在保持通用能力的基础上，学习遥感领域的特定知识；对于不同模态图像设计特定的低频信息输入模块，通过快速傅里叶变换为模型提供更多大尺度环境结构与上下文信息；利用基于注意力的高效特征融合模块，促进不同模态特征中相关信息的充分融合。在Vaihingen和Potsdam数据集上与15种方法进行评估比较，SAMLoF的mF1与mIoU指标均达到最优值，结果表明SAMLoF可有效提取并融合遥感图像中的不同模态特征，尤其能为复杂小目标物体生成精确且平滑的边界轮廓。

关键词: 遥感图像语义分割, 分割一切模型, 适配器微调, 低频信息, 特征融合

Abstract:

To address the problems of low small-target segmentation accuracy, insufficient cross-modal feature extraction and unsatisfactory fusion effect in multimodal remote sensing image semantic segmentation models, a dual-branch multimodal network Segment Anything Model Low-Frequency Fusion (SAMLoF), based on adapter optimization and low-frequency information extraction and fusion was proposed. By efficiently fine-tuning the parameters of the Segment Anything Model image encoder through the proposed joint adapter structure, the model could learn specific knowledge in the field of remote sensing while maintaining its general capabilities. The specific low-frequency information input modules for different modal images were designed to provide more large-scale environmental structures and contextual information for the model through the Fast Fourier Transform. An efficient feature fusion module based on attention was proposed to promote the full fusion of relevant information in different modal features. SAMLoF was evaluated and compared with 15 methods on the Vaihingen and Potsdam datasets. Both mF1 and mIoU metrics of SAMLoF reached the optimal values. The results showed that SAMLoF could effectively extract and fuse different modal features in remote sensing images, especially generating accurate and smooth boundary contours for complex small target objects.

Key words: semantic segmentation of remote sensing images, segment anything model, adapter fine-tuning low-frequency information, feature fusion

中图分类号:

TP751
TP391

李景涛, 封筠, 赵志宏. 适配器微调SAM与低频融合的遥感图像语义分割[J]. 图学学报, 2026, 47(3): 543-552.

LI Jingtao, FENG Jun, ZHAO Zhihong. Adapter fine-tuning SAM and low-frequency fusion for semantic segmentation of remote sensing images[J]. Journal of Graphics, 2026, 47(3): 543-552.

图/表 15

参考文献 28

[1]	WANG L B, LI R, DUAN C X, et al. A novel transformer based semantic segmentation scheme for fine-resolution remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 6506105.
[2]	郭丹青, 符颖, 朱烨, 等. 自注意力多尺度特征融合的遥感图像语义分割算法[J]. 计算机辅助设计与图形学学报, 2023, 35(8): 1259-1268.
	GUO D Q, FU Y, ZHU Y, et al. Semantic segmentation of remote sensing image via self-attention-based multi-scale feature fusion[J]. Journal of Computer-Aided Design & Computer Graphics, 2023, 35(8): 1259-1268(in Chinese).
[3]	CHEN J N, MEI J R, LI X H, et al. TransUNet: rethinking the u-net architecture design for medical image segmentation through the lens of transformers[J]. Medical Image Analysis, 2024, 97: 103280. DOI URL
[4]	WANG D, ZHANG J, DU B, et al. SAMRS: scaling-up remote sensing segmentation dataset with segment anything model[C]//The 37th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2023: 385.
[5]	KIRILLOV A, MINTUN E, RAVI N, et al. Segment anything[C]//IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 3992-4003.
[6]	MA X P, WU Q Q, ZHAO X Y, et al. SAM-assisted remote sensing imagery semantic segmentation with object and boundary constraints[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5636916.
[7]	张仕洁, 张斌, 赵文豪. 基于CNN和视觉状态空间的遥感影像语义分割[J]. 计算机应用研究, 2025, 42(5): 1583-1588.
	ZHANG S J, ZHANG B, ZHAO W H. CNN and visual state space based semantic segmentation of remote sensing images[J]. Application Research of Computers, 2025, 42(5): 1583-1588(in Chinese).
[8]	LIU Y, TIAN Y J, ZHAO Y Z, et al. VMamba: visual state space model[C]//The 38th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2024: 3273.
[9]	ZHU E Z, CHEN Z, WANG D K, et al. UNetMamba: an efficient UNet-like mamba for semantic segmentation of high-resolution remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2025, 22: 6001205.
[10]	HE S M, YANG H Q, ZHANG X Y, et al. MFTransNet: a multi-modal fusion with CNN-transformer network for semantic segmentation of HSR remote sensing images[J]. Mathematics, 2023, 11(3): 722. DOI URL
[11]	Ma X P, ZHANG X K, PUN M O, et al. A multilevel multimodal fusion transformer for remote sensing semantic segmentation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5403215.
[12]	李智杰, 程鑫, 李昌华, 等. 跨模态多层特征融合的遥感影像语义分割[J]. 计算机科学与探索, 2025, 19(4): 989-1000. DOI
	LI Z J, CHENG X, LI C H, et al. Cross-modal multi-level feature fusion for semantic segmentation of remote sensing images[J]. Journal of Frontiers of Computer Science and Technology, 2025, 19(4): 989-1000(in Chinese).
[13]	胡宇翔, 余长宏, 高明. 多模态融合的遥感图像语义分割网络[J]. 计算机工程与应用, 2024, 60(15): 234-242. DOI
	HU Y X, YU C H, GAO M. Remote sensing image semantic segmentation network based on multimodal fusion[J]. Computer Engineering and Applications, 2024, 60(15): 234-242(in Chinese). DOI
[14]	YAN Z Y, LI J X, LI X X, et al. RingMo-SAM: a foundation model for segment anything in multimodal remote-sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5625716.
[15]	ZHANG J L, ZHOU Z L, MAI G C, et al. Text2Seg: zero-shot remote sensing image semantic segmentation via text-guided visual foundation models[C]//The 7th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery. New York: ACM, 2024: 63-66.
[16]	ZHAO J Q, ZHANG M, ZHOU Z H, et al. CFFormer: a cross-fusion transformer framework for the semantic segmentation of multisource remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 4401117.
[17]	WANG L B, LI R, ZHANG C, et al. UNetFormer: a UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 190: 196-214. DOI URL
[18]	HOULSBY N, GIURGIU A, JASTRZEBSKI S, et al. Parameter- efficient transfer learning for NLP[EB/OL]. [2025-12-18]. https://proceedings.mlr.press/v97/houlsby19a.html.
[19]	MA X P, ZHANG X K, PUN M O, et al. MANet: fine-tuning segment anything model for multimodal remote sensing semantic segmentation[EB/OL]. (2025-12-16) [2025-12-18]. https://arxiv.org/pdf/2410.11160.
[20]	MA X P, ZHANG X K, PUN M O. A crossmodal multiscale fusion network for semantic segmentation of remote sensing data[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2022, 15: 3463-3474. DOI URL
[21]	LI R, ZHENG S Y, ZHANG C, et al. ABCNet: attentive bilateral contextual network for efficient semantic segmentation of fine-resolution remotely sensed imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 181: 84-98. DOI URL
[22]	LI R, ZHENG S Y, DUAN C X, et al. Multistage attention ResU-Net for semantic segmentation of fine-resolution remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 8009205.
[23]	ZHAO H S, SHI J P, QI X J, et al. Pyramid scene parsing network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 6230-6239.
[24]	MA X P, ZHANG X K, PUN M O. RS³Mamba: visual state space model for remote sensing image semantic segmentation[J]. IEEE Geoscience and Remote Sensing Letters, 2024, 21: 6011405.
[25]	HAZIRBAS C, MA L N, DOMOKOS C, et al. FuseNet: incorporating depth into semantic segmentation via fusion- based CNN architecture[C]//The 13th Asian Conference on Computer Vision on Computer Vision-ACCV 2016. Cham: Springer, 2016: 213-228.
[26]	AUDEBERT N, LE SAUX B, LEFÈVRE S. Beyond RGB: very high resolution urban remote sensing with multimodal deep networks[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2018, 140: 20-32. DOI URL
[27]	SEICHTER D, KÖHLER M, LEWANDOWSKI B, et al. Efficient RGB-D semantic segmentation for indoor scene analysis[C]//2021 IEEE International Conference on Robotics and Automation. New York: IEEE Press, 2021: 13525-13531.
[28]	HOSSEINPOUR H, SAMADZADEGAN F, JAVAN F D. CMGFNet: a deep cross-modal gated fusion network for building extraction from very high-resolution remote sensing images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 184: 96-115. DOI URL

Method	Backbone	OA						mF1	mIoU
Method	Backbone	Bui.	Tre.	Low.	Car	Imp.	Total	mF1	mIoU
ABCNet	ResNet18	94.10	90.81	78.53	64.12	89.70	89.25	85.34	75.20
MAResU-Net	ResNet18	94.84	89.99	79.09	85.89	92.19	90.17	88.54	79.89
PSPNet	ResNet101	94.52	90.17	78.84	79.22	92.03	89.94	86.55	76.96
UNetFormer	ResNet18	96.23	91.85	79.95	86.99	91.85	91.17	89.48	81.97
SAMRS	ResNet18	97.68	92.25	79.42	88.20	92.31	91.73	90.40	82.88
RS3Mamba	R18-Mamba-T	97.40	92.14	79.56	88.15	92.19	91.64	90.34	82.78
UNetMamba	ResT-Lite	98.05	92.34	80.93	88.19	93.15	92.24	90.62	83.28
FuseNet	VGG16	96.28	90.28	78.98	81.37	91.66	90.51	87.71	78.71
VFuseNet	VGG16	95.92	91.36	77.64	76.06	91.85	90.49	87.89	78.92
ESANet	ResNet34	95.69	90.50	77.16	85.46	91.39	90.61	88.18	79.42
CMGFNet	ResNet34	97.75	91.60	80.03	87.28	92.35	91.72	90.00	82.26
CMFNet	VGG16	97.17	90.82	80.37	85.47	92.36	91.40	89.48	81.44
MFTransNet	ResNet34	96.41	91.48	80.09	86.52	92.11	91.22	89.62	81.61
TransUNet	R50-ViT-B	96.48	92.77	76.14	69.56	91.66	90.96	87.34	78.26
FTransUNet	R50-ViT-B	98.20	91.94	81.49	91.27	93.01	92.40	91.21	84.23
SAMLoF	ViT-B	98.35	92.72	81.06	88.32	93.37	92.55	90.80	83.54
SAMLoF	ViT-L	98.77	93.34	81.24	89.65	93.47	92.96	91.69	84.92

Method	Backbone	OA						mF1	mIoU
Method	Backbone	Bui.	Tre.	Low.	Car	Imp.	Total	mF1	mIoU
ABCNet	ResNet18	94.10	90.81	78.53	64.12	89.70	89.25	85.34	75.20
MAResU-Net	ResNet18	94.84	89.99	79.09	85.89	92.19	90.17	88.54	79.89
PSPNet	ResNet101	94.52	90.17	78.84	79.22	92.03	89.94	86.55	76.96
UNetFormer	ResNet18	96.23	91.85	79.95	86.99	91.85	91.17	89.48	81.97
SAMRS	ResNet18	97.68	92.25	79.42	88.20	92.31	91.73	90.40	82.88
RS3Mamba	R18-Mamba-T	97.40	92.14	79.56	88.15	92.19	91.64	90.34	82.78
UNetMamba	ResT-Lite	98.05	92.34	80.93	88.19	93.15	92.24	90.62	83.28
FuseNet	VGG16	96.28	90.28	78.98	81.37	91.66	90.51	87.71	78.71
VFuseNet	VGG16	95.92	91.36	77.64	76.06	91.85	90.49	87.89	78.92
ESANet	ResNet34	95.69	90.50	77.16	85.46	91.39	90.61	88.18	79.42
CMGFNet	ResNet34	97.75	91.60	80.03	87.28	92.35	91.72	90.00	82.26
CMFNet	VGG16	97.17	90.82	80.37	85.47	92.36	91.40	89.48	81.44
MFTransNet	ResNet34	96.41	91.48	80.09	86.52	92.11	91.22	89.62	81.61
TransUNet	R50-ViT-B	96.48	92.77	76.14	69.56	91.66	90.96	87.34	78.26
FTransUNet	R50-ViT-B	98.20	91.94	81.49	91.27	93.01	92.40	91.21	84.23
SAMLoF	ViT-B	98.35	92.72	81.06	88.32	93.37	92.55	90.80	83.54
SAMLoF	ViT-L	98.77	93.34	81.24	89.65	93.47	92.96	91.69	84.92

Method	Backbone	OA						mF1	mIoU
Method	Backbone	Bui.	Tre.	Low.	Car	Imp.	Total	mF1	mIoU
ABCNet	ResNet18	96.23	78.92	86.40	92.92	88.90	87.52	88.14	79.26
MAResU-Net	ResNet18	96.82	83.97	87.70	95.88	92.19	89.82	90.86	83.61
PSPNet	ResNet101	97.03	83.13	85.67	88.81	90.91	88.67	88.92	80.36
UNetFormer	ResNet18	97.69	86.47	87.93	95.91	92.27	90.65	91.71	85.05
SAMRS	ResNet18	97.57	86.62	88.03	96.12	92.35	90.98	91.83	85.37
RS3Mamba	R18-Mamba-T	97.70	86.11	89.53	96.23	91.36	90.49	91.69	85.01
UNetMamba	ResT-Lite	97.98	87.34	87.86	96.05	92.43	91.12	92.05	85.57
FuseNet	VGG16	97.48	85.14	87.31	96.10	92.64	90.58	91.60	84.86
VFuseNet	VGG16	97.23	84.29	89.03	95.49	91.62	90.22	91.26	84.26
ESANet	ResNet34	97.10	85.31	87.81	94.08	92.76	89.74	91.22	84.15
CMGFNet	ResNet34	97.41	86.80	86.68	95.68	92.60	90.21	91.40	84.53
CMFNet	VGG16	97.63	87.40	88.00	95.68	92.84	91.16	92.10	85.63
MFTransNet	ResNet34	97.37	85.71	86.92	96.05	92.45	89.96	91.11	84.04
TransUNet	R50-ViT-B	96.63	82.65	89.98	93.17	91.93	90.01	90.97	83.74
FTransUNet	R50-ViT-B	97.78	88.27	88.48	96.31	93.17	91.34	92.41	86.20
SAMLoF	ViT-B	98.15	87.66	87.94	96.18	92.85	91.35	92.33	85.81
SAMLoF	ViT-L	98.33	88.82	88.38	96.37	93.75	91.66	92.68	86.55

Method	Backbone	OA						mF1	mIoU
Method	Backbone	Bui.	Tre.	Low.	Car	Imp.	Total	mF1	mIoU
ABCNet	ResNet18	96.23	78.92	86.40	92.92	88.90	87.52	88.14	79.26
MAResU-Net	ResNet18	96.82	83.97	87.70	95.88	92.19	89.82	90.86	83.61
PSPNet	ResNet101	97.03	83.13	85.67	88.81	90.91	88.67	88.92	80.36
UNetFormer	ResNet18	97.69	86.47	87.93	95.91	92.27	90.65	91.71	85.05
SAMRS	ResNet18	97.57	86.62	88.03	96.12	92.35	90.98	91.83	85.37
RS3Mamba	R18-Mamba-T	97.70	86.11	89.53	96.23	91.36	90.49	91.69	85.01
UNetMamba	ResT-Lite	97.98	87.34	87.86	96.05	92.43	91.12	92.05	85.57
FuseNet	VGG16	97.48	85.14	87.31	96.10	92.64	90.58	91.60	84.86
VFuseNet	VGG16	97.23	84.29	89.03	95.49	91.62	90.22	91.26	84.26
ESANet	ResNet34	97.10	85.31	87.81	94.08	92.76	89.74	91.22	84.15
CMGFNet	ResNet34	97.41	86.80	86.68	95.68	92.60	90.21	91.40	84.53
CMFNet	VGG16	97.63	87.40	88.00	95.68	92.84	91.16	92.10	85.63
MFTransNet	ResNet34	97.37	85.71	86.92	96.05	92.45	89.96	91.11	84.04
TransUNet	R50-ViT-B	96.63	82.65	89.98	93.17	91.93	90.01	90.97	83.74
FTransUNet	R50-ViT-B	97.78	88.27	88.48	96.31	93.17	91.34	92.41	86.20
SAMLoF	ViT-B	98.15	87.66	87.94	96.18	92.85	91.35	92.33	85.81
SAMLoF	ViT-L	98.33	88.82	88.38	96.37	93.75	91.66	92.68	86.55

EA	LFIM	EFM	Total	mF1	mIoU
			92.12	90.29	82.71
√			92.38	90.50	83.07
	√		92.25	90.46	82.98
		√	92.29	90.55	83.09
√	√		92.42	90.68	83.34
√	√	√	92.55	90.80	83.54

适配器微调SAM与低频融合的遥感图像语义分割

Adapter fine-tuning SAM and low-frequency fusion for semantic segmentation of remote sensing images

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 15

参考文献 28

相关文章 15

编辑推荐

Metrics

本文评价

EA	LFIM	EFM	Total	mF1	mIoU
			90.89	91.79	85.14
√			91.13	92.02	85.39
	√		91.12	92.06	85.45
		√	91.08	92.07	85.43
√	√		91.23	92.15	85.57
√	√	√	91.35	92.33	85.81

Modality	Method	OA						mF1	mIoU
Modality	Method	Bui.	Tre.	Low.	Car	Imp.	Total	mF1	mIoU
RGB	Baseline	96.08	92.45	81.50	86.04	92.08	91.52	90.00	82.15
RGB	SAMLoF	95.68	92.63	79.92	87.12	93.23	91.57	90.16	82.39
RGB+DSM	Baseline	97.90	91.66	80.98	87.08	93.04	92.12	90.29	82.71
RGB+DSM	SAMLoF	98.35	92.72	81.06	88.32	93.37	92.55	90.80	83.54

Method	FLOPs/G	Param/M	Memo/MB
CMGFNet	19.51	64.20	2463
CMFNet	78.25	123.63	4508
FTransUNet	69.85	203.40	3463
SAMLoF(ViT-B)	76.93	20.68	2416
SAMLoF(ViT-L)	284.93	57.16	5683

[1]	吴文欢, 王文舒, 王舒鳌. 融合层次化双流注意力的单目深度估计方法[J]. 图学学报, 2026, 47(3): 553-563.
[2]	房友江, 王世豪, 张亮, 段可然, 刘越, 魏小鹏, 杨鑫. 基于图拓扑特征提取的跨模态一致性检测方法[J]. 图学学报, 2026, 47(2): 286-295.
[3]	张行顺, 陈海永. 基于动态视觉传感器的航发叶片缺陷检测[J]. 图学学报, 2026, 47(1): 120-130.
[4]	樊乐翔, 马冀, 周登文. 基于退化分离的轻量级盲超分辨率重建网络[J]. 图学学报, 2025, 46(6): 1304-1315.
[5]	翟永杰, 翟邦朝, 胡哲东, 杨珂, 王乾铭, 赵晓瑜. 基于自适应特征融合金字塔与注意力机制的输电线路绝缘子缺陷检测方法[J]. 图学学报, 2025, 46(5): 950-959.
[6]	郭瑞东, 蓝贵文, 范冬林, 钟展, 徐梓睿, 任新月. 基于特征聚焦扩散网络的电力巡检目标检测算法[J]. 图学学报, 2025, 46(4): 719-726.
[7]	闫卓越, 刘骊, 付晓东, 刘利军, 彭玮. 三维人体姿态和形状估计的分层注意力时空特征融合算法[J]. 图学学报, 2025, 46(4): 746-755.
[8]	廖国琼, 黄龙杰, 李清新, 辜勇, 李海波. 面向单目可见光环境的自适应双手重建网络[J]. 图学学报, 2025, 46(4): 837-846.
[9]	董佳乐, 邓正杰, 李喜艳, 王诗韵. 基于频域和空域多特征融合的深度伪造检测方法[J]. 图学学报, 2025, 46(1): 104-113.
[10]	卢洋, 陈林慧, 姜晓恒, 徐明亮. SDENet：基于多尺度注意力质量感知的合成缺陷数据评价网络[J]. 图学学报, 2025, 46(1): 94-103.
[11]	闫建红, 冉同霄. 基于YOLOv8的轻量化无人机图像目标检测算法[J]. 图学学报, 2024, 45(6): 1328-1337.
[12]	吴沛宸, 袁立宁, 胡皓, 刘钊, 郭放. 基于注意力特征融合的视频异常行为检测[J]. 图学学报, 2024, 45(5): 922-929.
[13]	刘丽, 张起凡, 白宇昂, 黄凯烨. 结合Swin Transformer的多尺度遥感图像变化检测研究[J]. 图学学报, 2024, 45(5): 941-956.
[14]	章东平, 魏杨悦, 何数技, 徐云超, 胡海苗, 黄文君. 特征融合与层间传递：一种基于Anchor DETR改进的目标检测方法[J]. 图学学报, 2024, 45(5): 968-978.
[15]	罗智徽, 胡海涛, 马潇峰, 程文刚. 基于同质中间模态的跨模态行人再识别方法[J]. 图学学报, 2024, 45(4): 670-682.