Frequency-aware hypergraph fusion for event-based semantic segmentation

doi:10.11996/JG.j.2095-302X.2025061267

Abstract

Abstract:

Semantic segmentation, a core task for autonomous driving perception, faces challenges under low-light and high-speed scenarios due to the limitations of conventional cameras. Event cameras, with their microsecond temporal resolution and high dynamic range, effectively mitigate motion blur and extreme lighting conditions. However, their asynchronous sparse event data lacks texture and color information, while uneven event distributions caused by relative motion between background and objects pose significant difficulties for semantic feature extraction. To address these issues, a multi-frequency hypergraph fusion method for event-based semantic segmentation was proposed. First, the approach decomposed event frames into multi-scale spatiotemporal features through a frequency separation module, distinguishing high-frequency motion edges from low-frequency structural information. A dynamic hypergraph construction algorithm then mapped these multi-frequency features into hypergraph nodes, utilizing hypergraph convolution to capture long-range dependencies across frequencies. Finally, an attention mechanism adaptively fused multi-frequency features to enhance inter-class discriminability. Experiments on Carla-Semantic and DDD17-Semantic datasets demonstrated that this method achieved 88.21% MPA and 82.68% mIoU, outperforming existing methods and validating the effectiveness of the multi-frequency hypergraph model for event-based semantic understanding. This research provided a novel solution for robust environment perception with event cameras, particularly suited to challenging autonomous driving scenarios involving low-light conditions and rapid motion.

Key words: semantic segmentation, hypergraph, attention mechanism, multi-frequency fusion, event camera

CLC Number:

YU Nannan, MENG Zhengyu, FANG Youjiang, SUN Chuanyu, YIN Xuefeng, ZHANG Qiang, WEI Xiaopeng, YANG Xin. Frequency-aware hypergraph fusion for event-based semantic segmentation[J]. Journal of Graphics, 2025, 46(6): 1267-1273.

Figures/Tables 6

Fig. 1 A framework for event semantic segmentation method integrating multi frequency hypergraphs

Fig. 2 Dynamic multi frequency fusion multi-frequency hypergraph

Table 1 Comparison of quantitative indicators in carla semantic dataset/%

方法	MPA	mIoU
FCN	80.86	72.00
UNet	82.89	74.87
PSPNet	84.36	76.35
DeepLab v3+	81.53	75.03
E2VID	51.50	39.59
Ev-SegNet	86.04	80.25
VID2E	87.12	80.71
SETR	81.77	73.78
SegFormer	85.66	79.75
Swin-UperNet	86.61	80.73
EVISS	87.89	81.93
AFFormer	87.36	81.54
Ours	88.21	82.68

Fig. 3 Comparison of visualization results on the Carla-Semantic dataset ((a) RGB frame; (b) Event frame; (c) Ground-truth labels; (d) Swin-UperNet; (e) EVISS; (f) Ours)

Table 2 Comparison of quantitative indicators in DDD17-Semantic dataset/%

方法	mIoU
E2VID	44.77±3.70
Ev-SegNet	54.81
VID2E	56.01
EVISS	57.64
AFFormer	52.46
Ours	58.95

Table 3 Quantitative indicators of ablation experiment

方法	DMFF	MFHGA	MPA/%	mIoU/%
Ours			78.81	71.54
Ours	√		79.44	72.35
Ours		√	87.56	81.73
Ours	√	√	88.21	82.68

References 27

[1]	ALONSO I, MURILLO A C. EV-SegNet: semantic segmentation for event-based cameras[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2019: 1624-1633.
[2]	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1800-1807.
[3]	GEHRIG D, GEHRIG M, HIDALGO-CARRIÓ J, et al. Video to events: recycling video datasets for event cameras[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 3583-3592.
[4]	REBECQ H, RANFTL R, KOLTUN V, et al. High speed and high dynamic range video with an event camera[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(6): 1964-1980. DOI URL
[5]	MUNDA G, REINBACHER C, POCK T. Real-time intensity-image reconstruction for event cameras using manifold regularisation[J]. International Journal of Computer Vision, 2018, 126(12): 1381-1393. DOI
[6]	HU Y H, DELBRUCK T, LIU S C. Learning to exploit multiple vision modalities by using grafted networks[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 85-101.
[7]	王超毅, 于男男, 乔羽, 等. 基于事件相机的图像语义分割方法[EB/OL]. [2024-10-08]. http://kns.cnki.net/kcms/detail/11.2925.tp.20241008.1600.005.html.
	WANG C Y, YU N N, QIAO Y, et al. Event-based image semantic segmentation[EB/OL]. [2024-10-08]. http://kns.cnki.net/kcms/detail/11.2925.tp.20241008.1600.005.html. (in Chinese).
[8]	HUANG Y C, LIU Q S, METAXAS D. Video object segmentation by hypergraph cut[C]// 2009 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2009: 1738-1745.
[9]	HUANG Y C, LIU Q S, ZHANG S T, et al. Image retrieval via probabilistic hypergraph ranking[C]// 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2010: 3376-3383.
[10]	GAO Y, WANG M, TAO D C, et al. 3-D object retrieval and recognition with hypergraph analysis[J]. IEEE Transactions on Image Processing, 2012, 21(9): 4290-4303. DOI PMID
[11]	ZHOU D Y, HUANG J Y, SCHÖLKOPF B. Learning with hypergraphs: clustering, classification, and embedding[C]// The 20th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2006: 1601-1608.
[12]	CHEN F H, GAO Y, CAO D L, et al. Multimodal hypergraph learning for microblog sentiment prediction[C]// 2015 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2015: 1-6.
[13]	JI R R, CHEN F H, CAO L J, et al. Cross-modality microblog sentiment prediction via bi-layer multimodal hypergraph learning[J]. IEEE Transactions on Multimedia, 2019, 21(4): 1062-1075. DOI URL
[14]	ZHANG Z Z, LIN H J, GAO Y. Dynamic hypergraph structure learning[EB/OL]. [2024-08-09]. https://www.ijcai.org/Proceedings/2018/0439.pdf
[15]	FENG Y F, YOU H X, ZHANG Z Z, et al. Hypergraph neural networks[C]// The 33rd AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2019: 3558-3565.
[16]	ZHONG Y J, LI B, TANG L, et al. Detecting camouflaged object in frequency domain[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 4494-4503.
[17]	QIAN Y Y, YIN G J, SHENG L, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 86-103.
[18]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[EB/OL]. (2021-06-03) [2024-07-10]. https://arxiv.org/abs/2010.11929.
[19]	BINAS J, NEIL D, LIU S C, et al. DDD17:end-to-end DAVIS driving dataset[EB/OL]. (2017-11-04) [2024-07-10]. https://arxiv.org/abs/1711.01458.
[20]	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]// The 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[21]	ZHAO H S, SHI J P, QI X J, et al. Pyramid scene parsing network[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 6230-6239.
[22]	CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder- decoder with atrous separable convolution for semantic image segmentation[C]// The 15th European Conference on Computer Vision. Cham: Springer, 2018: 833-851.
[23]	ZHENG S X, LU J C, ZHAO H S, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 6877-6886.
[24]	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 9992-10002.
[25]	XIE E Z, WANG W H, YU Z D, et al. SegFormer: simple and efficient design for semantic segmentation with transformers[C]// The 35th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2021: 12077-12090.
[26]	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 3431-3440.
[27]	DONG B, WANG P C, WANG F. Head-free lightweight semantic segmentation with linear transformer[C]// The 37th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2023: 516-524.