Multimodal Beat-STMAN network with beat alignment for dance motion recognition

doi:10.11996/JG.j.2095-302X.2026030534

Abstract

Abstract:

To address insufficient spatiotemporal feature coupling and the inadequate utilization of multimodal information in dance motion recognition, a multi-modal fusion Beat STMAN (Beat-guided Spatio-Temporal Multimodal AMAA-Net Network) network recognition method was proposed to improve the accuracy of dance action recognition. Firstly, this method is based on the skeleton based ST-GCN (Spatial Temporal Graph Convolutional Network) model to construct a spatiotemporal convolutional skeleton network. To cope with the unfavorable factors of continuous and varied dance movements and partially obscured movements, a dynamic adjacency matrix was integrated with a multi-head spatial attention mechanism to automatically capture global human-body topology parameters. Secondly, an audio stream information feature extraction alignment fusion method was proposed to obtain beat timestamp pulse sequences, and a Transformer multi-head attention mechanism was used to design a cross-modal fusion module AMAA-Net, which achieved multimodal feature fusion through a resistance game mechanism, effectively alleviating insufficient model feature fusion. Finally, the Beat-STMAN was evaluated on a publicly available dance dataset to verify its effectveness. Experimental results showed that for the Thomas spin movement, the recognition rate of the proposed model achieved 14.7% higher than that of the ST-GCN model, demonstration significantly improved robustness in occluded scenarios. Furthermore, ablation experiments verified that the integration of the dynamic adjacency matrix, multi-head attention mechanism, and cross-modal attention mechanism could effectively fuse audio-action correlation features, with a cross-modal contribution rate reaching 5.3%. This effectively improved the model’s Top-1 accuracy, thereby enhancing the model’s prediction precision and providing a multimodal technical implementation path for dance-teaching evaluation and immersive interaction.

Key words: dance motion recognition, Beat-STMAN, multimodal attention, Transformer, multi-mode

CLC Number:

TANG Haiying, LI Fang. Multimodal Beat-STMAN network with beat alignment for dance motion recognition[J]. Journal of Graphics, 2026, 47(3): 534-542.

Figures/Tables 8

References 38

[1]	雷建云, 梁钧, 夏梦, 等. 融合时空注意力的改进ST-GCN人体动作识别方法研究[J]. 中南民族大学学报(自然科学版), 2025, 44(4): 526-535.
	LEI J Y, LIANG J, XIA M, et al. Research on the improved ST-GCN method for human action recognition by integrating spatiotemporal attention[J]. Journal of South-Central Minzu University (Natural Science Edition), 2025, 44(4): 526-535(in Chinese).
[2]	张琼. 融合注意力机制的图神经网络运动动作识别研究[J]. 自动化与仪器仪表, 2025(3): 196-200.
	ZHANG Q. Research on motor action recognition based on graph neural network integrating attention mechanism[J]. Automation & Instrumentation, 2025(3): 196-200(in Chinese).
[3]	魏欣然, 何昕怡, 杨秀秀, 等. 基于改进Transformer模型的电工手部动作识别与分类方法[J]. 湖北民族大学学报(自然科学版), 2025, 43(2): 253-258.
	WEI X R, HE X Y, YANG X X, et al. Method for recognizing and classifying electricians’ hand movements based on improved transformer model[J]. Journal of Hubei Minzu University (Natural Science Edition), 2025, 43(2): 253-258(in Chinese).
[4]	LIU J, SHAHROUDY A, XU D, et al. Skeleton-based action recognition using spatio-temporal LSTM network with trust gates[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(12): 3007-3021. DOI PMID
[5]	TRAN D, BOURDEV L, FERGUS R, et al. Learning spatiotemporal features with 3D convolutional networks[C]// 2015 IEEE International Conference on Computer Vision. New York: IEEE Press, 2015: 4489-4497.
[6]	DONAHUE J, HENDRICKS L A, ROHRBACH M, et al. Long-term recurrent convolutional networks for visual recognition and description[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 677-691. DOI PMID
[7]	蒋悦晗, 陈俊杰, 李洪均. 基于骨骼图神经网络的人体行为识别综述[J]. 计算机工程与应用, 2025, 61(3): 34-47. DOI
	JIANG Y H, CHEN J J, LI H J. Review of human action recognition based on skeletal graph neural networks[J]. Computer Engineering and Applications, 2025, 61(3): 34-47(in Chinese). DOI
[8]	JI S W, XU W, YANG M, et al. 3D convolutional neural networks for human action recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1): 221-231. PMID
[9]	顾瑞坤. 融合2D与3D信息的特定舞蹈姿态识别算法设计[J]. 电子设计工程, 2025, 33(9): 144-148.
	GU R K. Design of a specific dance posture recognition algorithm by integrating 2D and 3D information[J]. Electronic Design Engineering, 2025, 33(9): 144-148(in Chinese).
[10]	许诚, 金庆红. 基于多特征融合的复杂舞蹈动作识别[J]. 安徽工程大学学报, 2022, 37(3): 64-69.
	XU C, JIN Q H. Recognition of complex dance movements based on multi feature fusion[J]. Journal of Anhui Polytechnic University, 2022, 37(3): 64-69(in Chinese).
[11]	YANG X Q. Enhancing research on dance action recognition by integrating collaborative CAD through multimodal fusion[J]. Computer-Aided Design & Applications, 2025, 22(S5): 229-243.
[12]	FANG Z, ZHANG X W, CAO T Y, et al. A new adjacency matrix configuration in GCN-based models for skeleton-based action recognition[EB/OL]. [2025-04-24]. https://arxiv.org/abs/2206.14344.
[13]	代金利, 曹江涛, 姬晓飞. 交互关系超图卷积模型的双人交互行为识别[J]. 智能系统学报, 2024, 19(2): 316-324.
	DAI J L, CAO J T, JI X F. Two-person interaction recognition based on the interactive relationship hypergraph convolution network model[J]. CAAI Transactions on Intelligent Systems, 2024, 19(2): 316-324(in Chinese).
[14]	刘钰, 铁有福, 江张军, 等. 基于图神经网络的人体动作识别方法[J]. 天津理工大学学报, 2025, 41(6): 73-79.
	LIU Y, TIE Y F, JIANG Z J, et al. Human action recognition method based on graph neural network[J]. Journal of Tianjin University of Technology, 2025, 41(6): 73-79(in Chinese).
[15]	杨红红, 王刘丽, 张玉梅, 等. 基于序列多尺度特征融合表示的层级舞蹈动作姿态估计方法[J]. 电子学报, 2021, 49(12): 2428-2436. DOI
	YANG H H, WANG L L, ZHANG Y M, et al. Hierarchical dance pose estimation algorithm based on sequential multi-scale feature fusion[J]. Acta Electronica Sinica, 2021, 49(12): 2428-2436(in Chinese).
[16]	黄攀, 张宇. 基于动作捕捉传感器的民族舞蹈动作自动识别系统[J]. 自动化与仪器仪表, 2022(8): 267-271, 276.
	HUANG P, ZHANG Y. Automatic recognition system of national dance movement based on motion capture sensor[J]. Automation & Instrumentation, 2022(8): 267-271, 276(in Chinese).
[17]	ZHOU Y X, YAN X D, CHENG Z Q, et al. BlockGCN: redefine topology awareness for skeleton-based action recognition[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 2049-2058.
[18]	LEE J, LEE M, LEE D, et al. Hierarchically decomposed graph convolutional networks for skeleton-based action recognition[C]//IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 10410-10419.
[19]	SHI Z S, LIANG J, LI Q Q, et al. Multi-modal multi-action video recognition[C]//IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 13658-13667.
[20]	SUN S K, LIU D Z, DONG J F, et al. Unified multi-modal unsupervised representation learning for skeleton-based action understanding[C]//The 31st ACM International Conference on Multimedia. New York: ACM, 2023: 2973-2984.
[21]	YAN S J, XIONG Y J, LIN D H. Spatial temporal graph convolutional networks for skeleton-based action recogni-tion[EB/OL]. [2025-04-24]. https://ojs.aaai.org/index.php/AAAI/article/view/12328.
[22]	李前, 杨文柱, 陈向阳, 等. 基于紧耦合时空双流卷积神经网络的人体动作识别模型[J]. 计算机应用, 2020, 40(11): 3178-3183. DOI
	LI Q, YANG W Z, CHEN X Y, et al. Human action recognition model based on tightly coupled spatiotemporal two-stream convolution neural network[J]. Journal of Computer Applications, 2020, 40(11): 3178-3183(in Chinese). DOI
[23]	ZHONG Q B, ZHENG C M, ZHANG H X. Research on discriminative skeleton-based action recognition in spatiotemporal fusion and human-robot interaction[J]. Complexity, 2020, 2020(1): 8717942.
[24]	HU K, JIN J L, ZHENG F, et al. Overview of behavior recognition based on deep learning[J]. Artificial Intelligence Review, 2022, 56(3): 1833-1865. DOI
[25]	杨世强, 李卓, 王金华, 等. 基于新分区策略的ST-GCN人体动作识别[J]. 计算机集成制造系统, 2023, 29(12): 4040-4050.
	YANG S Q, LI Z, WANG J H, et al. ST-GCN human action recognition based on new partition strategy[J]. Computer Integrated Manufacturing Systems, 2023, 29(12): 4040-4050(in Chinese).
[26]	冒鑫鑫, 吴胜昔, 咸博龙, 等. 基于骨架的自适应图卷积和LSTM行为识别[J]. 华东理工大学学报(自然科学版), 2022, 48(6): 816-825.
	MAO X X, WU S X, XIAN B L, et al. Adaptive graph convolution and LSTM action recognition based on skeleton[J]. Journal of East China University of Science and Technology, 2022, 48(6): 816-825(in Chinese).
[27]	WU F H F, LEE T C, JANG J S R, et al. A two-fold dynamic programming approach to beat tracking for audio music with time-varying tempo[EB/OL]. [2025-02-24]. https://dblp.uni-trier.de/db/conf/ismir/ismir2011.html#conf/ismir/WuLJCLW11.
[28]	许铭, 王冬霞, 周城旭, 等. 改进的Kullback-Leibler复非负矩阵分解语音增强算法[J]. 声学技术, 2019, 38(5): 560-567.
	XU M, WANG D X, ZHOU C X, et al. Speech enhancement based on improved Kullback-Leibler complex non-negative matrix factorization[J]. Technical Acoustics, 2019, 38(5): 560-567(in Chinese).
[29]	SHI Z F, WU Q B, MENG F M, et al. Cross-modal cognitive consensus guided audio-visual segmentation[J]. IEEE Transactions on Multimedia, 2025, 27: 209-223. DOI URL
[30]	HERSHEY S, CHAUDHURI S, ELLIS D P W, et al. CNN architectures for large-scale audio classification[C]//2017 IEEE International Conference on Acoustics, Speech and Signal Processing. New York: IEEE Press, 2017: 131-135.
[31]	田青, 虞静静, 张正. 结合自适应局部图卷积与多尺度时间建模的骨架行为识别[J]. 计算机应用研究, 2025, 42(7): 2199-2205.
	TIAN Q, YU J J, ZHANG Z. Skeleton behavior recognition combining adaptive local graph convolution with multi-scale time modeling[J]. Application Research of Computers, 2025, 42(7): 2199-2205(in Chinese).
[32]	KIM H, CHOI Y, KIM J, et al. Exploiting spatial dimensions of latent in GAN for real-time image editing[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 852-861.
[33]	LI Z Q, TAO R T, WANG J, et al. Interpreting the latent space of GANs via measuring decoupling[J]. IEEE Transactions on Artificial Intelligence, 2021, 2(1): 58-70. DOI URL
[34]	赵杰, 郭东. 基于平行注意力机制的对抗样本防御方法[J]. 吉林大学学报(信息科学版), 2022, 40(5): 846-855.
	ZHAO J, GUO D. Adversarial examples defense method base on parallel attention mechanism[J]. Journal of Jilin University (Information Science Edition), 2022, 40(5): 846-855(in Chinese).
[35]	WANG T, JIN L, WANG Z, et al. SynSP: synergy of smoothness and precision in pose sequences refinement[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 1824-1833.
[36]	LI R L, YANG S, ROSS D A, et al. Learn to dance with AIST++:music conditioned 3D dance generation[EB/OL]. [2025-04-24]. https://arxiv.org/abs/2101.08779v1.
[37]	OBINATA Y, YAMAMOTO T. Temporal extension module for skeleton-based action recognition[C]//2020 25th International Conference on Pattern Recognition. New York: IEEE Press, 2021: 534-540.
[38]	SHI L, ZHANG Y F, CHENG J, et al. Skeleton-based action recognition with multi-stream adaptive graph convolutional networks[J]. IEEE Transactions on Image Processing, 2020, 29: 9532-9545. DOI URL

序列标注(截取部分)	类别	序列标注(截取部分)	类别
	Lock Step		Headspin
	Thomas		Shake
	Robot		Bounce
	Floating		Whacks

序列标注(截取部分)	类别	序列标注(截取部分)	类别
	Lock Step		Headspin
	Thomas		Shake
	Robot		Bounce
	Floating		Whacks

模型	组件	Top-1准确率	Top-5准确率	ΔTop-1
模型1	静态拓扑、AMAA-Net	83.5±2.3	92.1±0.9	-7.7
模型2	动态单头、AMAA-Net	85.6±1.7	95.1±0.3	-5.6
模型3	动态多头、线性拼接	85.9±1.5	94.2±1.1	-5.3
模型4	动态多头、AMAA-Net	91.2±0.8	98.5±0.3	─

模型	组件	Top-1准确率	Top-5准确率	ΔTop-1
模型1	静态拓扑、AMAA-Net	83.5±2.3	92.1±0.9	-7.7
模型2	动态单头、AMAA-Net	85.6±1.7	95.1±0.3	-5.6
模型3	动态多头、线性拼接	85.9±1.5	94.2±1.1	-5.3
模型4	动态多头、AMAA-Net	91.2±0.8	98.5±0.3	─

模型	AIST_Part数据集
模型	Top-1准确率	Top-5准确率
ST-GCN	83.4±7.3	93.1±0.3
2s-AGCN	85.7±0.5	94.3±1.7
MST-GCN	88.6±2.1	96.5±1.3
2s-AGCN+TEM^[37]	89.5±1.9	97.4±2.2
DGNN^[38]	87.7±2.3	95.6±1.5
HD-GCN^[18]	89.0±2.0	96.3±1.4
BlockGCN^[17]	90.6±1.8	97.2±1.2
MMAVR^[19]	90.5±1.7	97.5±1.1
Beat-STMAN	91.2±2.8	98.5±0.3