基于注意力特征融合的视频异常行为检测

doi:10.11996/JG.j.2095-302X.2024050922

图学学报 ›› 2024, Vol. 45 ›› Issue (5): 922-929.DOI: 10.11996/JG.j.2095-302X.2024050922

• 图像处理与计算机视觉 • 上一篇下一篇

基于注意力特征融合的视频异常行为检测

吴沛宸¹(), 袁立宁²^,³, 胡皓¹, 刘钊⁴, 郭放¹()

1.中国人民公安大学信息网络安全学院，北京 100038
2.中国人民公安大学国家安全学院，北京 100038
3.广西警察学院信息技术学院，广西南宁 530028
4.中国人民公安大学网络空间安全与法治协同创新中心，北京 100038

收稿日期:2024-05-08 修回日期:2024-06-25 出版日期:2024-10-31 发布日期:2024-10-31
通讯作者:郭放(1988-)，女，讲师，博士。主要研究方向为视频图像技术、计算机视觉。E-mail：guofang@ppsuc.edu.cn
第一作者:吴沛宸(1997-)，男，硕士研究生。主要研究方向为计算机视觉。E-mail：m13209406252@163.com
基金资助:
中国人民公安大学安全防范工程双一流创新研究专项(2023SYL08);广西哲学社会科学研究课题项目(23FTQ005);广西壮族自治区公安厅专项课题(2023GAQN092)

Video anomaly detection based on attention feature fusion

WU Peichen¹(), YUAN Lining²^,³, HU Hao¹, LIU Zhao⁴, GUO Fang¹()

1. College of Information and Network Security, People’s Public Security University of China, Beijing 100038, China
2. School of National Security, People’s Public Security University of China, Beijing 100038, China
3. School of Information Technology, Guangxi Police College, Nanning Guangxi 530028, China
4. Collaborative Innovation Center for Network Security and Rule of Law, People’s Public Security University of China, Beijing 100038, China

Received:2024-05-08 Revised:2024-06-25 Published:2024-10-31 Online:2024-10-31
Contact: GUO Fang (1988-), lecturer, Ph.D. Her main research interests cover the technology of video and computer vision.
E-mail：guofang@ppsuc.edu.cn
First author：WU Peichen (1997-), master student. His main research interest covers computer vision. E-mail：m13209406252@163.com
Supported by:
Double First-Class Innovation Research Project for People’s Public Security University of China(2023SYL08);The Social Science Fund of Guangxi(23FTQ005);Public Security Department Project of Guangxi(2023GAQN092)

摘要/Abstract

摘要：

当前以多头自注意力为代表的基于注意力机制的特征融合方法在很大程度上取决于特征间的相关性，其跨域融合能力有限，且特征融合后由于不同特征的域间差异，导致时空感知能力不足，有效融合2种跨域特征仍面临挑战。针对RGB特征和光流特征跨域表达能力不足、融合后特征的时空感知能力弱等问题，提出了一种基于注意力特征融合的视频异常行为检测方法。首先采用一种轻量级注意力特征融合模块(LAFF)构筑融合机制，进行RGB和光流特征的融合，进而在增强融合后特征表达能力的同时减少网络参数量，提高异常检测算法性能。在全局时空感知阶段，通过多分支卷积模块(DBB)增强特征时空感知能力，同时需兼顾计算复杂度和检测效果。在UCSD Ped2数据集上取得了99.85%的识别效果，在CUHK Avenue和LAD 2000数据集上表现同样良好，验证了该方法的有效性。

关键词: 计算机视觉, 异常行为检测, 特征融合, 注意力机制, 多分支卷积

Abstract:

Currently, feature fusion methods based on attention mechanisms, such as multi-head self-attention, largely rely on the correlation between features, with limited cross-domain fusion capabilities. Additionally, due to existing domain differences among various features, the spatiotemporal perception capability of the fused features is insufficient. To address the insufficient cross-domain expression capability of RGB and optical flow features and the weak spatiotemporal perception capability of the fused features, a video anomaly detection method based on attentional feature fusion was proposed. Firstly, a lightweight attentional feature fusion module (LAFF) was employed to construct the fusion mechanism, combining RGB and optical flow features, enhancing the feature expression capabilities while reducing the network’s parameter count and improving anomaly detection algorithm performance. Then, in the global spatiotemporal perception stage, a diverse branch block (DBB) was utilized to enhance the spatiotemporal perception capabilities of the features, while considering computational complexity and detection effectiveness. Finally, the proposed method achieved a recognition rate of 99.85% on the UCSD Ped2 dataset and demonstrated similarly strong performance on the CUHK Avenue and LAD 2000 datasets, validating the effectiveness of the approach.

Key words: computer vision, video anomaly detection, feature fusion, attention mechanism, diverse branch block

中图分类号:

TP391.41

吴沛宸, 袁立宁, 胡皓, 刘钊, 郭放. 基于注意力特征融合的视频异常行为检测[J]. 图学学报, 2024, 45(5): 922-929.

WU Peichen, YUAN Lining, HU Hao, LIU Zhao, GUO Fang. Video anomaly detection based on attention feature fusion[J]. Journal of Graphics, 2024, 45(5): 922-929.

图/表 9

图1 异常行为检测算法框架图

Fig. 1 Framework of anomaly detection algorithm

图2 LAFF特征融合流程图

Fig. 2 LAFF feature fusion process flowchart

图3 多分支卷积模块DBB网络结构图

Fig. 3 Multi-branch convolution module DBB network structure diagram

图4 DBB模块在推理阶段的6种转换方式

Fig. 4 Six transformation methods of the DBB module in the inference stage

图5 AUC对比图

Fig. 5 AUC comparison chart ((a) LAD2000; (b) UCSD Ped2; (c) CUHK Avenue)

表1 消融实验结果

Table 1 Ablation experiment results

Fusion Method	AUC/%			Param/M	FLOPs/G
Fusion Method	CUHK Avenue	UCSD Ped2	LAD2000	Param/M	FLOPs/G
Concat^[8]	86.44	93.83	85.25	-	-
Concat+MS-CAM^[13]	56.64	51.57	46.58	4×D×d+d²	17.29
Concat+SENet^[20]	87.47	93.72	86.49	2×D×d	1.03
Concat+MHSA^[21]	86.41	94.70	85.93	D×d+4×d²	94.90
LAFF	89.91	98.92	87.00	D×d	0.61

表2 使用不同改进方法时模型性能表现/%

Table 2 Model performance with different improvement methods/%

Method	AUC
Method	CUHK Avenue	UCSD Ped2	LAD 2000
Baseline	86.44	93.83	85.25
Baseline+LAFF	89.91	98.92	87.00
Baseline+DBB	86.48	94.57	85.93
Baseline+LAFF+DBB	90.21	99.85	87.54

表3 与现有方法对比结果/%

Table 3 Comparison results with existing methods/%

Method	CUHK Avenue	UCSD Ped2	LAD2000
AST-AE^[2]	85.20	96.60	-
文献[3]	87.10	96.70	-
文献[4]	89.10	-	-
DeepMIL^[5]	87.53	90.19	70.18
AR-Net^[6]	89.31	93.64	79.84
MLEP^[7]	89.20	-	50.57
文献[8]	89.33	95.12	86.28
LFP_MBA^[14]	89.80	99.40	-
文献[17]	88.50	-	-
MNAD^[22]	82.80	90.20	45.84
STC-Net^[23]	87.80	96.70	-
Ours	90.21	99.85	87.54

图6 LAD2000数据集异常行为检测效果图((a)拥挤类异常；(b)破坏类异常)

Fig. 6 Effectiveness chart for anomaly detection in LAD2000 dataset ((a) Congestion-type anomalies; (b) Destruction-type anomalies)

参考文献 23

[1]	徐海涛. 基于深度学习的视频异常行为检测研究[D]. 北京: 北京交通大学, 2022.
	XU H T. Research on video anomalous behavior detection based on deep learning[D]. Beijing: Beijing Jiaotong University, 2022 (in Chinese).
[2]	LIU Y, YANG D K, WANG Y, et al. Generalized video anomaly event detection: systematic taxonomy and comparison of deep models[EB/OL]. [2024-03-11]. http://arxiv.org/abs/2302.05087.
[3]	CHANG Y P, TU Z G, XIE W, et al. Video anomaly detection with spatio-temporal dissociation[J]. Pattern Recognition, 2022, 122: 108213.
[4]	吕浩, 易鹏飞, 刘瑞, 等. 用于视频异常检测的时序多尺度自编码器[J]. 图学学报, 2022, 43(2): 223-229.
	LYU H, YI P F, LIU R, et al. Sequential multi-scale autoencoder for video anomaly detection[J]. Journal of Graphics, 2022, 43(2): 223-229 (in Chinese). DOI
[5]	SULTANI W, CHEN C, SHAH M. Real-world anomaly detection in surveillance videos[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 6479-6488.
[6]	WAN B Y, FANG Y M, XIA X, et al. Weakly supervised video anomaly detection via center-guided discriminative learning[C]// 2020 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2020: 1-6.
[7]	LIU W, LUO W, LI Z, et al. Margin learning embedded prediction for video anomaly detection with a few anomalies[C]// IJCAI’19: The 28th International Joint Conference on Artificial Intelligence. New York: ACM, 2019: 3023-3030.
[8]	WAN B Y, JIANG W H, FANG Y M, et al. Anomaly detection in video sequences: a benchmark and computational model[J]. IET Image Processing, 2021, 15(14): 3454-3465.
[9]	肖进胜, 申梦瑶, 江明俊, 等. 融合包注意力机制的监控视频异常行为检测[J]. 自动化学报, 2022, 48(12): 2951-2959.
	XIAO J S, SHEN M Y, JIANG M J, et al. Abnormal behavior detection algorithm with video-bag attention mechanism in surveillance video[J]. Acta Automatica Sinica, 2022, 48(12): 2951-2959 (in Chinese).
[10]	ZHOU Y Z, SUN X Y, LUO C, et al. Spatiotemporal fusion in 3D CNNs: a probabilistic view[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 9826-9835.
[11]	张和为. 基于特征融合网络的视频人脸识别技术研究[D]. 桂林: 桂林电子科技大学, 2022.
	ZHANG H W. Research on video face recognition technology based on feature fusion network[D]. Guilin: Guilin University of Electronic Technology, 2022 (in Chinese).
[12]	程相贵, 刘钊, 郭放. 结合双流I3D和注意力机制的视频异常事件检测[J]. 信息与电脑: 理论版, 2022, 34(24): 65-68.
	CHENG X G, LIU Z, GUO F. Video anomaly event detection combining dual-stream I3D and attention mechanism[J]. Information & Computer: Theory Edition, 2022, 34(24): 65-68 (in Chinese).
[13]	MA X, JI Z X, NIU S J, et al. MS-CAM: multi-scale class activation maps for weakly-supervised segmentation of geographic atrophy lesions in SD-OCT images[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(12): 3443-3455.
[14]	黄少年, 文沛然, 全琪, 等. 基于多支路聚合的帧预测轻量化视频异常检测[J]. 图学学报, 2023, 44(6): 1173-1182. DOI
	HUANG S N, WEN P R, QUAN Q, et al. Future frame prediction based on multi-branch aggregation for lightweight video anomaly detection[J]. Journal of Graphics, 2023, 44(6): 1173-1182 (in Chinese).
[15]	DING X H, ZHANG X Y, HAN J G, et al. Diverse branch block: building a convolution as an inception-like unit[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 10881-10890.
[16]	SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 1-9.
[17]	闫善武, 肖洪兵, 王瑜, 等. 融合行人时空信息的视频异常检测[J]. 图学学报, 2023, 44(1): 95-103. DOI
	YAN S W, XIAO H B, WANG Y, et al. Video anomaly detection combining pedestrian spatiotemporal information[J]. Journal of Graphics, 2023, 44(1): 95-103 (in Chinese).
[18]	ROKA S, DIWAKAR M. DSLSTM: a deep convolutional encoder-decoder architecture for abnormality detection in video surveillance[J]. Cluster Computing, 2024(17): 4925-4940.
[19]	LI K C, WANG Y L, HE Y N, et al. UniFormerV2: unlocking the potential of image ViTs for video understanding[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 1632-1643.
[20]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7132-7141.
[21]	LV H, YUE Z Q, SUN Q R, et al. Unbiased multiple instance learning for weakly supervised video anomaly detection[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 8022-8031.
[22]	PARK H, NOH J, HAM B. Learning memory-guided normality for anomaly detection[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 14360-14369.
[23]	ZHAO M Y, LIU Y, LIU J, et al. Exploiting spatial-temporal correlations for video anomaly detection[C]// 2022 26th International Conference on Pattern Recognition. New York: IEEE Press, 2022: 1727-1733.

基于注意力特征融合的视频异常行为检测

Video anomaly detection based on attention feature fusion

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 23

相关文章 15

编辑推荐

Metrics

本文评价

[1]	刘冀辰, 李金星, 吴佳, 张威, 齐宇诺, 周国亮. 大模型技术在电力行业的应用展望[J]. 图学学报, 2024, 45(6): 1132-1144.
[2]	李琼 , 考月英 , 张莹 , 徐沛 . 面向无人机航拍图像的目标检测研究综述[J]. 图学学报, 2024, 45(6): 1145-1164.
[3]	闫建红, 冉同霄. 基于YOLOv8的轻量化无人机图像目标检测算法[J]. 图学学报, 2024, 45(6): 1328-1337.
[4]	胡凤阔, 叶兰, 谭显峰, 张钦展, 胡志新, 方清, 王磊, 满孝锋. 一种基于改进YOLOv8的轻量化路面病害检测算法[J]. 图学学报, 2024, 45(5): 892-900.
[5]	刘义艳, 郝婷楠, 贺晨, 常英杰. 基于DBBR-YOLO的光伏电池表面缺陷检测[J]. 图学学报, 2024, 45(5): 913-921.
[6]	刘丽, 张起凡, 白宇昂, 黄凯烨. 结合Swin Transformer的多尺度遥感图像变化检测研究[J]. 图学学报, 2024, 45(5): 941-956.
[7]	章东平, 魏杨悦, 何数技, 徐云超, 胡海苗, 黄文君. 特征融合与层间传递：一种基于Anchor DETR改进的目标检测方法[J]. 图学学报, 2024, 45(5): 968-978.
[8]	谢国波, 林松泽, 林志毅, 吴陈锋, 梁立辉. 基于改进YOLOv7-tiny的道路病害检测算法[J]. 图学学报, 2024, 45(5): 987-997.
[9]	熊超, 王云艳, 罗雨浩. 特征对齐与上下文引导的多视图三维重建[J]. 图学学报, 2024, 45(5): 1008-1016.
[10]	彭文, 林金炜. 基于空间信息关注和纹理增强的短小染色体分类方法[J]. 图学学报, 2024, 45(5): 1017-1029.
[11]	刘宗明, 洪唯, 龙睿, 祝越, 张小宇. 基于自注意机制的乳源瑶绣自动生成与应用研究[J]. 图学学报, 2024, 45(5): 1096-1105.
[12]	李大湘, 吉展, 刘颖, 唐垚. 改进YOLOv7遥感图像目标检测算法[J]. 图学学报, 2024, 45(4): 650-658.
[13]	罗智徽, 胡海涛, 马潇峰, 程文刚. 基于同质中间模态的跨模态行人再识别方法[J]. 图学学报, 2024, 45(4): 670-682.
[14]	魏敏, 姚鑫. 基于多尺度与注意力机制的两阶段风暴单体外推研究[J]. 图学学报, 2024, 45(4): 696-704.
[15]	牛为华, 郭迅. 基于改进YOLOv8的船舰遥感图像旋转目标检测算法[J]. 图学学报, 2024, 45(4): 726-735.