Video anomaly detection based on attention feature fusion

doi:10.11996/JG.j.2095-302X.2024050922

Abstract

Abstract:

Currently, feature fusion methods based on attention mechanisms, such as multi-head self-attention, largely rely on the correlation between features, with limited cross-domain fusion capabilities. Additionally, due to existing domain differences among various features, the spatiotemporal perception capability of the fused features is insufficient. To address the insufficient cross-domain expression capability of RGB and optical flow features and the weak spatiotemporal perception capability of the fused features, a video anomaly detection method based on attentional feature fusion was proposed. Firstly, a lightweight attentional feature fusion module (LAFF) was employed to construct the fusion mechanism, combining RGB and optical flow features, enhancing the feature expression capabilities while reducing the network’s parameter count and improving anomaly detection algorithm performance. Then, in the global spatiotemporal perception stage, a diverse branch block (DBB) was utilized to enhance the spatiotemporal perception capabilities of the features, while considering computational complexity and detection effectiveness. Finally, the proposed method achieved a recognition rate of 99.85% on the UCSD Ped2 dataset and demonstrated similarly strong performance on the CUHK Avenue and LAD 2000 datasets, validating the effectiveness of the approach.

Key words: computer vision, video anomaly detection, feature fusion, attention mechanism, diverse branch block

CLC Number:

TP391.41

WU Peichen, YUAN Lining, HU Hao, LIU Zhao, GUO Fang. Video anomaly detection based on attention feature fusion[J]. Journal of Graphics, 2024, 45(5): 922-929.

Figures/Tables 9

References 23

[1]	徐海涛. 基于深度学习的视频异常行为检测研究[D]. 北京: 北京交通大学, 2022.
	XU H T. Research on video anomalous behavior detection based on deep learning[D]. Beijing: Beijing Jiaotong University, 2022 (in Chinese).
[2]	LIU Y, YANG D K, WANG Y, et al. Generalized video anomaly event detection: systematic taxonomy and comparison of deep models[EB/OL]. [2024-03-11]. http://arxiv.org/abs/2302.05087.
[3]	CHANG Y P, TU Z G, XIE W, et al. Video anomaly detection with spatio-temporal dissociation[J]. Pattern Recognition, 2022, 122: 108213.
[4]	吕浩, 易鹏飞, 刘瑞, 等. 用于视频异常检测的时序多尺度自编码器[J]. 图学学报, 2022, 43(2): 223-229.
	LYU H, YI P F, LIU R, et al. Sequential multi-scale autoencoder for video anomaly detection[J]. Journal of Graphics, 2022, 43(2): 223-229 (in Chinese). DOI
[5]	SULTANI W, CHEN C, SHAH M. Real-world anomaly detection in surveillance videos[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 6479-6488.
[6]	WAN B Y, FANG Y M, XIA X, et al. Weakly supervised video anomaly detection via center-guided discriminative learning[C]// 2020 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2020: 1-6.
[7]	LIU W, LUO W, LI Z, et al. Margin learning embedded prediction for video anomaly detection with a few anomalies[C]// IJCAI’19: The 28th International Joint Conference on Artificial Intelligence. New York: ACM, 2019: 3023-3030.
[8]	WAN B Y, JIANG W H, FANG Y M, et al. Anomaly detection in video sequences: a benchmark and computational model[J]. IET Image Processing, 2021, 15(14): 3454-3465.
[9]	肖进胜, 申梦瑶, 江明俊, 等. 融合包注意力机制的监控视频异常行为检测[J]. 自动化学报, 2022, 48(12): 2951-2959.
	XIAO J S, SHEN M Y, JIANG M J, et al. Abnormal behavior detection algorithm with video-bag attention mechanism in surveillance video[J]. Acta Automatica Sinica, 2022, 48(12): 2951-2959 (in Chinese).
[10]	ZHOU Y Z, SUN X Y, LUO C, et al. Spatiotemporal fusion in 3D CNNs: a probabilistic view[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 9826-9835.
[11]	张和为. 基于特征融合网络的视频人脸识别技术研究[D]. 桂林: 桂林电子科技大学, 2022.
	ZHANG H W. Research on video face recognition technology based on feature fusion network[D]. Guilin: Guilin University of Electronic Technology, 2022 (in Chinese).
[12]	程相贵, 刘钊, 郭放. 结合双流I3D和注意力机制的视频异常事件检测[J]. 信息与电脑: 理论版, 2022, 34(24): 65-68.
	CHENG X G, LIU Z, GUO F. Video anomaly event detection combining dual-stream I3D and attention mechanism[J]. Information & Computer: Theory Edition, 2022, 34(24): 65-68 (in Chinese).
[13]	MA X, JI Z X, NIU S J, et al. MS-CAM: multi-scale class activation maps for weakly-supervised segmentation of geographic atrophy lesions in SD-OCT images[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(12): 3443-3455.
[14]	黄少年, 文沛然, 全琪, 等. 基于多支路聚合的帧预测轻量化视频异常检测[J]. 图学学报, 2023, 44(6): 1173-1182. DOI
	HUANG S N, WEN P R, QUAN Q, et al. Future frame prediction based on multi-branch aggregation for lightweight video anomaly detection[J]. Journal of Graphics, 2023, 44(6): 1173-1182 (in Chinese).
[15]	DING X H, ZHANG X Y, HAN J G, et al. Diverse branch block: building a convolution as an inception-like unit[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 10881-10890.
[16]	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 1-9.
[17]	闫善武, 肖洪兵, 王瑜, 等. 融合行人时空信息的视频异常检测[J]. 图学学报, 2023, 44(1): 95-103. DOI
	YAN S W, XIAO H B, WANG Y, et al. Video anomaly detection combining pedestrian spatiotemporal information[J]. Journal of Graphics, 2023, 44(1): 95-103 (in Chinese).
[18]	ROKA S, DIWAKAR M. DSLSTM: a deep convolutional encoder-decoder architecture for abnormality detection in video surveillance[J]. Cluster Computing, 2024(17): 4925-4940.
[19]	LI K C, WANG Y L, HE Y N, et al. UniFormerV2: unlocking the potential of image ViTs for video understanding[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 1632-1643.
[20]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7132-7141.
[21]	LV H, YUE Z Q, SUN Q R, et al. Unbiased multiple instance learning for weakly supervised video anomaly detection[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 8022-8031.
[22]	PARK H, NOH J, HAM B. Learning memory-guided normality for anomaly detection[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 14360-14369.
[23]	ZHAO M Y, LIU Y, LIU J, et al. Exploiting spatial-temporal correlations for video anomaly detection[C]// 2022 26th International Conference on Pattern Recognition. New York: IEEE Press, 2022: 1727-1733.

Fusion Method	AUC/%			Param/M	FLOPs/G
Fusion Method	CUHK Avenue	UCSD Ped2	LAD2000	Param/M	FLOPs/G
Concat^[8]	86.44	93.83	85.25	-	-
Concat+MS-CAM^[13]	56.64	51.57	46.58	4×D×d+d²	17.29
Concat+SENet^[20]	87.47	93.72	86.49	2×D×d	1.03
Concat+MHSA^[21]	86.41	94.70	85.93	D×d+4×d²	94.90
LAFF	89.91	98.92	87.00	D×d	0.61

Fusion Method	AUC/%			Param/M	FLOPs/G
Fusion Method	CUHK Avenue	UCSD Ped2	LAD2000	Param/M	FLOPs/G
Concat^[8]	86.44	93.83	85.25	-	-
Concat+MS-CAM^[13]	56.64	51.57	46.58	4×D×d+d²	17.29
Concat+SENet^[20]	87.47	93.72	86.49	2×D×d	1.03
Concat+MHSA^[21]	86.41	94.70	85.93	D×d+4×d²	94.90
LAFF	89.91	98.92	87.00	D×d	0.61

Method	AUC
Method	CUHK Avenue	UCSD Ped2	LAD 2000
Baseline	86.44	93.83	85.25
Baseline+LAFF	89.91	98.92	87.00
Baseline+DBB	86.48	94.57	85.93
Baseline+LAFF+DBB	90.21	99.85	87.54

Method	AUC
Method	CUHK Avenue	UCSD Ped2	LAD 2000
Baseline	86.44	93.83	85.25
Baseline+LAFF	89.91	98.92	87.00
Baseline+DBB	86.48	94.57	85.93
Baseline+LAFF+DBB	90.21	99.85	87.54

Method	CUHK Avenue	UCSD Ped2	LAD2000
AST-AE^[2]	85.20	96.60	-
文献[3]	87.10	96.70	-
文献[4]	89.10	-	-
DeepMIL^[5]	87.53	90.19	70.18
AR-Net^[6]	89.31	93.64	79.84
MLEP^[7]	89.20	-	50.57
文献[8]	89.33	95.12	86.28
LFP_MBA^[14]	89.80	99.40	-
文献[17]	88.50	-	-
MNAD^[22]	82.80	90.20	45.84
STC-Net^[23]	87.80	96.70	-
Ours	90.21	99.85	87.54