欢迎访问《图学学报》 分享到：
图学学报

图学学报 ›› 2021, Vol. 42 ›› Issue (3): 439-445.DOI: 10.11996/JG.j.2095-302X.2021030439

• 图像处理与计算机视觉 • 上一篇    下一篇

融合非局部神经网络的行为检测模型 

  

  1. 桂林电子科技大学计算机与信息安全学院，广西 桂林 541004
  • 出版日期:2021-06-30 发布日期:2021-06-29
  • 基金资助:
    广西图像图形智能处理重点实验室培育基地(桂林电子科技大学)开放基金项目(GIIP2011)

Action detection model fused with non-local neural network

  1. School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin Guangxi 541004, China
  • Online:2021-06-30 Published:2021-06-29
  • Supported by:
    Open Funds from Guilin University of Electronic Technology, Guangxi Key Laboratory of Image and Graphic Intelligent Processing (GIIP2011)

PDF (PC)

81

摘要： 针对在视频行为检测中卷积神经网络(CNN)对时域信息理解能力不足的问题，提出了一种融合 非局部神经网络的行为检测模型。模型采用一种双分支的 CNN 结构，分别提取视频的空间特征和运动特征。 将视频单帧和视频连续帧序列作为网络输入，空间网络对视频当前帧进行 2D CNN 特征提取，时空网络采用融 合非局部模块的 3D CNN 来捕获视频帧之间的全局联系。为了进一步增强上下文语义信息，使用一种通道融合 机制来聚合双分支网络的特征，最后将融合后的特征用于帧级检测。在 UCF101-24 和 JHMDB 2 个数据集上进 行了实验，结果表明，该方法能够充分融合空间和时间维度信息，在基于视频的时空行为检测任务上具有较高 的检测精度。

关键词: 行为检测, 非局部模块, 3D 卷积, 注意力机制

Abstract: The convolutional neural network (CNN) has insufficient ability to understand the time domain information in video action detection. For this problem, we proposed a model based on fused non-local neural network, which combines non-local block with 3D CNN to capture global connections between video frames. Model used a two-stream architecture of 2D CNN and 3D CNN to extract the spatial and motion features of the video, respectively, which takes video single frames and video frame sequences as inputs. To further enhance contextual semantic information, an improved attention and channel fusion mechanism is used to aggregate the features of the above two networks, and finally the fused features are used for frame-level detection. We conducted experimental verification and comparison on the UCF101-24 and JHMDB data set. The results show that our method can fully integrate spatial and temporal information, and has high detection accuracy on video-based action detection tasks. 

Key words: action detection, non-local neural network, 3D convolution, attention mechanism 

中图分类号: