融合多频超图的图像语义分割方法

doi:10.11996/JG.j.2095-302X.2025061267

图学学报 ›› 2025, Vol. 46 ›› Issue (6): 1267-1273.DOI: 10.11996/JG.j.2095-302X.2025061267

• 图像处理与计算机视觉 • 上一篇下一篇

融合多频超图的图像语义分割方法

于男男¹^,²(), 孟政宇¹^,², 房友江¹^,², 孙传昱¹^,², 殷雪峰¹^,², 张强¹^,², 魏小鹏¹^,², 杨鑫¹^,²()

¹ 大连理工大学计算机科学与技术学院，辽宁大连 116024
² 社会计算与认知智能教育部重点实验室，辽宁大连 116024

收稿日期:2024-10-09 接受日期:2025-04-16 出版日期:2025-12-30 发布日期:2025-12-27
通讯作者:杨鑫(1984-)，男，教授，博士。主要研究方向为计算机图形学与计算机视觉等。E-mail：xinyang@dlut.edu.cn
第一作者:于男男(1993-)，女，博士研究生。主要研究方向为基于事件相机的计算机视觉。E-mail：12009059@mail.dlut.edu.cn
基金资助:
科技创新2030——“新一代人工智能”重大项目(2021ZD12400)

Frequency-aware hypergraph fusion for event-based semantic segmentation

YU Nannan¹^,²(), MENG Zhengyu¹^,², FANG Youjiang¹^,², SUN Chuanyu¹^,², YIN Xuefeng¹^,², ZHANG Qiang¹^,², WEI Xiaopeng¹^,², YANG Xin¹^,²()

¹ School of Computer Science and Technology, Dalian University of Technology, Dalian Liaoning 116024, China
² Key Laboratory of Social Computing and Cognitive Intelligence of Ministry of Education, Dalian Liaoning 116024, China

Received:2024-10-09 Accepted:2025-04-16 Published:2025-12-30 Online:2025-12-27
First author：YU Nannan (1993-), PhD candidate. Her main research interest covers event-based computer vision. E-mail：12009059@mail.dlut.edu.cn
Supported by:
Science and Technology Innovation 2030 - “New Generation Artificial Intelligence” Major Project(2021ZD12400)

摘要/Abstract

摘要：

语义分割技术作为自动驾驶环境感知的核心任务，在复杂光照和高速运动场景下面临着传统相机成像质量不足的挑战。事件相机凭借微秒级时间分辨率和高动态范围特性，能够有效克服运动模糊和极端光照条件，但其异步稀疏事件数据缺乏纹理和色彩信息，且背景和目标运动导致事件分布不均匀，为语义特征提取带来显著困难。针对这些问题，提出了一种融合多频超图的事件图像语义分割方法。首先通过频率分解模块提取事件帧的多尺度时空特征，以分离高频运动边缘与低频结构信息；继而设计动态超图构建算法，将不同频率特征映射为超图节点，利用超图卷积捕获跨频率的长程依赖关系，最后通过注意力机制自适应融合特征，增强类别间的判别性。在Carla-Semantic和DDD17-Semantic数据集上的实验表明，该方法在MPA(88.21%)和mIoU(82.68%)指标上优于现有事件分割方法，验证了多频超图模型对事件数据语义理解的提升效果。为基于事件相机的鲁棒性环境感知提供了新思路，特别适用于自动驾驶中的低光照、高速运动等挑战场景。

关键词: 语义分割, 超图, 注意力机制, 多频融合, 事件相机

Abstract:

Semantic segmentation, a core task for autonomous driving perception, faces challenges under low-light and high-speed scenarios due to the limitations of conventional cameras. Event cameras, with their microsecond temporal resolution and high dynamic range, effectively mitigate motion blur and extreme lighting conditions. However, their asynchronous sparse event data lacks texture and color information, while uneven event distributions caused by relative motion between background and objects pose significant difficulties for semantic feature extraction. To address these issues, a multi-frequency hypergraph fusion method for event-based semantic segmentation was proposed. First, the approach decomposed event frames into multi-scale spatiotemporal features through a frequency separation module, distinguishing high-frequency motion edges from low-frequency structural information. A dynamic hypergraph construction algorithm then mapped these multi-frequency features into hypergraph nodes, utilizing hypergraph convolution to capture long-range dependencies across frequencies. Finally, an attention mechanism adaptively fused multi-frequency features to enhance inter-class discriminability. Experiments on Carla-Semantic and DDD17-Semantic datasets demonstrated that this method achieved 88.21% MPA and 82.68% mIoU, outperforming existing methods and validating the effectiveness of the multi-frequency hypergraph model for event-based semantic understanding. This research provided a novel solution for robust environment perception with event cameras, particularly suited to challenging autonomous driving scenarios involving low-light conditions and rapid motion.

Key words: semantic segmentation, hypergraph, attention mechanism, multi-frequency fusion, event camera

中图分类号:

于男男, 孟政宇, 房友江, 孙传昱, 殷雪峰, 张强, 魏小鹏, 杨鑫. 融合多频超图的图像语义分割方法[J]. 图学学报, 2025, 46(6): 1267-1273.

YU Nannan, MENG Zhengyu, FANG Youjiang, SUN Chuanyu, YIN Xuefeng, ZHANG Qiang, WEI Xiaopeng, YANG Xin. Frequency-aware hypergraph fusion for event-based semantic segmentation[J]. Journal of Graphics, 2025, 46(6): 1267-1273.

图/表 6

图1 融合多频超图的事件语义分割方法框架

Fig. 1 A framework for event semantic segmentation method integrating multi frequency hypergraphs

图2 动态多频滤波融合多频超图

Fig. 2 Dynamic multi frequency fusion multi-frequency hypergraph

表1 Carla-Semantic数据集量化指标对比/%

Table 1 Comparison of quantitative indicators in carla semantic dataset/%

方法	MPA	mIoU
FCN	80.86	72.00
UNet	82.89	74.87
PSPNet	84.36	76.35
DeepLab v3+	81.53	75.03
E2VID	51.50	39.59
Ev-SegNet	86.04	80.25
VID2E	87.12	80.71
SETR	81.77	73.78
SegFormer	85.66	79.75
Swin-UperNet	86.61	80.73
EVISS	87.89	81.93
AFFormer	87.36	81.54
Ours	88.21	82.68

图3 Carla-Semantic数据集上可视化结果对比((a) RGB图像；(b) 事件帧图像；(c) 真值标签；(d) Swin-UperNet；(e) EVISS；(f) 本文方法)

Fig. 3 Comparison of visualization results on the Carla-Semantic dataset ((a) RGB frame; (b) Event frame; (c) Ground-truth labels; (d) Swin-UperNet; (e) EVISS; (f) Ours)

表2 DDD17-Semantic数据集量化指标对比/%

Table 2 Comparison of quantitative indicators in DDD17-Semantic dataset/%

方法	mIoU
E2VID	44.77±3.70
Ev-SegNet	54.81
VID2E	56.01
EVISS	57.64
AFFormer	52.46
Ours	58.95

表3 消融实验量化指标

Table 3 Quantitative indicators of ablation experiment

方法	DMFF	MFHGA	MPA/%	mIoU/%
Ours			78.81	71.54
Ours	√		79.44	72.35
Ours		√	87.56	81.73
Ours	√	√	88.21	82.68

参考文献 27

[1]	ALONSO I, MURILLO A C. EV-SegNet: semantic segmentation for event-based cameras[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2019: 1624-1633.
[2]	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1800-1807.
[3]	GEHRIG D, GEHRIG M, HIDALGO-CARRIÓ J, et al. Video to events: recycling video datasets for event cameras[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 3583-3592.
[4]	REBECQ H, RANFTL R, KOLTUN V, et al. High speed and high dynamic range video with an event camera[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(6): 1964-1980. DOI URL
[5]	MUNDA G, REINBACHER C, POCK T. Real-time intensity-image reconstruction for event cameras using manifold regularisation[J]. International Journal of Computer Vision, 2018, 126(12): 1381-1393. DOI
[6]	HU Y H, DELBRUCK T, LIU S C. Learning to exploit multiple vision modalities by using grafted networks[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 85-101.
[7]	王超毅, 于男男, 乔羽, 等. 基于事件相机的图像语义分割方法[EB/OL]. [2024-10-08]. http://kns.cnki.net/kcms/detail/11.2925.tp.20241008.1600.005.html.
	WANG C Y, YU N N, QIAO Y, et al. Event-based image semantic segmentation[EB/OL]. [2024-10-08]. http://kns.cnki.net/kcms/detail/11.2925.tp.20241008.1600.005.html. (in Chinese).
[8]	HUANG Y C, LIU Q S, METAXAS D. Video object segmentation by hypergraph cut[C]// 2009 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2009: 1738-1745.
[9]	HUANG Y C, LIU Q S, ZHANG S T, et al. Image retrieval via probabilistic hypergraph ranking[C]// 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2010: 3376-3383.
[10]	GAO Y, WANG M, TAO D C, et al. 3-D object retrieval and recognition with hypergraph analysis[J]. IEEE Transactions on Image Processing, 2012, 21(9): 4290-4303. DOI PMID
[11]	ZHOU D Y, HUANG J Y, SCHÖLKOPF B. Learning with hypergraphs: clustering, classification, and embedding[C]// The 20th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2006: 1601-1608.
[12]	CHEN F H, GAO Y, CAO D L, et al. Multimodal hypergraph learning for microblog sentiment prediction[C]// 2015 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2015: 1-6.
[13]	JI R R, CHEN F H, CAO L J, et al. Cross-modality microblog sentiment prediction via bi-layer multimodal hypergraph learning[J]. IEEE Transactions on Multimedia, 2019, 21(4): 1062-1075. DOI URL
[14]	ZHANG Z Z, LIN H J, GAO Y. Dynamic hypergraph structure learning[EB/OL]. [2024-08-09]. https://www.ijcai.org/Proceedings/2018/0439.pdf
[15]	FENG Y F, YOU H X, ZHANG Z Z, et al. Hypergraph neural networks[C]// The 33rd AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2019: 3558-3565.
[16]	ZHONG Y J, LI B, TANG L, et al. Detecting camouflaged object in frequency domain[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 4494-4503.
[17]	QIAN Y Y, YIN G J, SHENG L, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 86-103.
[18]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[EB/OL]. (2021-06-03) [2024-07-10]. https://arxiv.org/abs/2010.11929.
[19]	BINAS J, NEIL D, LIU S C, et al. DDD17:end-to-end DAVIS driving dataset[EB/OL]. (2017-11-04) [2024-07-10]. https://arxiv.org/abs/1711.01458.
[20]	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]// The 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[21]	ZHAO H S, SHI J P, QI X J, et al. Pyramid scene parsing network[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 6230-6239.
[22]	CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder- decoder with atrous separable convolution for semantic image segmentation[C]// The 15th European Conference on Computer Vision. Cham: Springer, 2018: 833-851.
[23]	ZHENG S X, LU J C, ZHAO H S, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 6877-6886.
[24]	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 9992-10002.
[25]	XIE E Z, WANG W H, YU Z D, et al. SegFormer: simple and efficient design for semantic segmentation with transformers[C]// The 35th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2021: 12077-12090.
[26]	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 3431-3440.
[27]	DONG B, WANG P C, WANG F. Head-free lightweight semantic segmentation with linear transformer[C]// The 37th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2023: 516-524.

融合多频超图的图像语义分割方法

Frequency-aware hypergraph fusion for event-based semantic segmentation

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献 27

相关文章 15

编辑推荐

Metrics

本文评价

[1]	薄文, 琚晨, 刘维青, 张焱, 胡晶晶, 程婧晗, 张常有. 基于退化感知时序建模的装备维保时机预测方法[J]. 图学学报, 2025, 46(6): 1233-1246.
[2]	张馨匀, 张力文, 周李, 罗笑南. 基于图像分块交互的咖啡果实成熟度预测模型[J]. 图学学报, 2025, 46(6): 1274-1280.
[3]	肖凯, 袁玲, 储珺. 基于周期一致性和动态记忆增强的无监督无人机目标跟踪[J]. 图学学报, 2025, 46(6): 1281-1291.
[4]	曹璐静, 鲁鹏. 一种基于多参考图的视频上色方法[J]. 图学学报, 2025, 46(6): 1316-1326.
[5]	刘圆圆, 房友江, 孟天宇, 孟政宇, 罗鹏伟, 杨培根, 姜雨彤, 魏小鹏, 张强, 杨鑫. 基于几何超图感知的三维场景图生成[J]. 图学学报, 2025, 46(6): 1337-1345.
[6]	刘伯凯, 殷雪峰, 孙传昱, 葛慧林, 魏子麒, 姜雨彤, 朴海音, 周东生, 杨鑫. 基于深度强化学习的无人机三维场景导航方法研究[J]. 图学学报, 2025, 46(5): 1010-1017.
[7]	左屿琪, 张云峰, 张秋悦, 徐英城. 基于超图表示学习和Transformer模型优化的知识感知推荐[J]. 图学学报, 2025, 46(5): 1050-1060.
[8]	翟永杰, 翟邦朝, 胡哲东, 杨珂, 王乾铭, 赵晓瑜. 基于自适应特征融合金字塔与注意力机制的输电线路绝缘子缺陷检测方法[J]. 图学学报, 2025, 46(5): 950-959.
[9]	朱泓淼, 钟国杰, 张严辞. 基于均值漂移与深度学习融合的小语义点云语义分割[J]. 图学学报, 2025, 46(5): 998-1009.
[10]	陈东, 李昌隆, 杜振龙, 宋爽, 李晓丽. 光影智绘：基于SAM的视频阴影鲁棒抽取[J]. 图学学报, 2025, 46(4): 739-745.
[11]	杨佳熙, 于乐天, 包骐瑞, 毕胜, 麻晓斗, 杨晟琦, 姜雨彤, 方建儒, 魏小鹏, 杨鑫. 面向高光子通量环境的目标深度估计方法[J]. 图学学报, 2025, 46(4): 756-762.
[12]	牛杭, 葛鑫雨, 赵晓瑜, 杨珂, 王乾铭, 翟永杰. 基于改进YOLOv8的防振锤缺陷目标检测算法[J]. 图学学报, 2025, 46(3): 532-541.
[13]	崔丽莎, 宋志文, 姜晓恒, 马鑫, 陈恩庆, 徐明亮. 基于边界和语义感知的表面缺陷分割网络[J]. 图学学报, 2025, 46(3): 578-587.
[14]	于冰, 程广, 黄东晋, 丁友东. 基于双流网络融合的三维人体网格重建[J]. 图学学报, 2025, 46(3): 625-634.
[15]	雷玉林, 刘利刚. 基于深度强化学习的可缓冲的物体运输和装箱[J]. 图学学报, 2025, 46(3): 697-708.