基于离散自回归序列建模的实时语音驱动三维说话人动画生成

doi:10.11996/JG.j.2095-302X.2026040757

摘要/Abstract

摘要：

语音驱动的三维人脸动画生成是计算机视觉与计算机图形学中的重要研究方向，在虚拟数字人、实时交互系统以及沉浸式媒体等应用场景中具有广泛需求。该任务旨在根据输入语音信号生成与语音内容、节奏和发音特征高度一致的三维人脸运动，但现有方法往往依赖复杂的生成过程或多步采样机制，导致推理延迟较高，难以满足实时流式应用的性能要求。为解决上述问题，提出了一种面向实时生成的语音驱动三维人脸动画方法，将人脸运动建模为基于离散表示的自回归序列生成过程。首先利用VQ-VAE将连续的人脸运动参数映射到紧凑的离散token空间，从而降低序列建模难度，并在此基础上采用类语言模型的自回归Transformer，在语音条件约束下对离散人脸运动序列进行逐帧预测，实现低延迟、可流式的人脸动画生成。在人脸表示方面，统一采用FLAME2023参数化模型，并基于VHAP tracker对音视频数据进行重新拟合，以获得高质量且时间一致的人脸运动参数序列。实验结果表明，该方法在唇部同步、面部表情动态以及头部运动一致性等方面均取得了具有竞争力的表现，同时在推理效率和实时性能上显著优于现有生成方法，验证了其在实时交互场景中的实用性。相关研究为构建高效、可实时交互的三维虚拟人系统提供了一种可行且可扩展的技术路径，并为后续在离散自回归框架下进一步引入风格、情绪状态等高层语义控制因素奠定了基础。

关键词: 三维说话人视频生成, 实时流式推理, 离散自回归生成, 三维人脸参数化模型, 虚拟数字人

Abstract:

Speech-driven 3D facial animation generation is an important research topic in computer vision and computer graphics, with broad demand in applications such as virtual digital humans, real-time interactive systems, and immersive media. The goal of this task is to generate 3D facial motions that are highly consistent with the content, rhythm, and articulation characteristics of the input speech signal. However, existing methods often rely on complex generation pipelines or multi-step sampling mechanisms, resulting in high inference latency and making them difficult to deploy in real-time streaming scenarios. To address these limitations, a real-time-oriented approach for speech-driven 3D facial animation was proposed, in which facial motion generation was formulated as an autoregressive sequence modeling problem based on discrete representations. Specifically, continuous facial motion parameters were first mapped into a compact discrete token space using VQ-VAE, thereby reducing the difficulty of sequence modeling. On this basis, a language-model-like autoregressive Transformer was employed to predict discrete facial motion tokens frame by frame under speech-conditioned constraints, enabling low-latency and streamable facial animation generation. For facial representation, the FLAME2023 parametric model was adopted in a unified manner, and audio-visual data were re-fitted using the VHAP tracker to obtain high-quality and temporally consistent facial motion parameter sequences. Experimental results demonstrated that this approach achieved competitive performance in lip synchronization, facial-expression dynamics, and head-motion consistency, while significantly outperforming existing generation methods in terms of inference efficiency and real-time performance, validating its practical applicability in real-time interactive scenarios. This research provides a feasible and scalable technical pathway for building efficient, real-time interactive 3D virtual human systems, and lays the groundwork for further incorporation of higher-level semantic controls such as speaking style and emotional states within a discrete autoregressive framework.

Key words: 3D talking face video generation, real-time streaming inference, discrete autoregressive generation, parametric 3D face model, virtual digital humans

中图分类号:

TP391

周禹, 吕天, 李明, 刘永进. 基于离散自回归序列建模的实时语音驱动三维说话人动画生成[J]. 图学学报, 2026, 47(4): 757-765.

ZHOU Yu, LV Tian, LI Ming, LIU Yongjin. Real-time speech-driven 3D talking face animation via discrete autoregressive sequence modeling[J]. Journal of Graphics, 2026, 47(4): 757-765.

图/表 5

图1 基于离散自回归序列建模的实时语音驱动三维说话人动画生成

Fig. 1 Real-time speech-driven 3D talking face animation via discrete autoregressive sequence modeling

图2 离散人脸运动建模方法

Fig. 2 Discrete facial motion modeling method

表1 不同方法的生成质量评估

Table 1 Evaluation of generation quality across different methods

方法	LVE/mm↓	FDD/×10^-5m↓	FPS↑
FaceFormer^[4]	10.07	16.85	2.07
SelfTalk^[6]	12.03	15.93	23.91
ARTalk^[11]	10.31	11.01	18.31
DiffPoseTalk^[9]	8.83	10.46	0.14
本文方法	10.12	10.97	27.63

图3 定性实验结果

Fig. 3 Qualitative experimental results

表2 消融实验结果

Table 2 Ablation study

方法	LVE/mm↓	FDD/×10^-5m↓
本文方法(使用FLAME2020)	11.04	11.41
本文方法(去除平滑损失)	10.21	11.09
本文方法	10.12	10.97

参考文献 25

[1]	CHU X G, LI Y, ZENG A L, et al. GPAvatar: generalizable and precise head avatar from image(s)[EB/OL]. [2024-01-18]. http://arxiv.org/abs/2401.10215.
[2]	MA S J, WENG Y L, SHAO T J, et al. 3D Gaussian blendshapes for head avatar animation[C]// 2024 ACM SIGGRAPH Conference Papers. New York: ACM, 2024: 1-10.
[3]	LI T Y, BOLKART T, BLACK M J, et al. Learning a model of facial shape and expression from 4D scans[J]. ACM Transactions on Graphics, 2017, 36(6): 194.
[4]	FAN Y R, LIN Z J, SAITO J, et al. FaceFormer: speech-driven 3D facial animation with transformers[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 18770-18780.
[5]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30.
[6]	PENG Z Q, LUO Y H, SHI Y, et al. SelfTalk: a self-supervised commutative training diagram to comprehend 3D talking faces[C]// The 31st ACM International Conference on Multimedia. New York: ACM, 2023: 5292-5301.
[7]	HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[J]. Advances in Neural Information Processing Systems, 2020, 33: 6840-6851.
[8]	SONG J M, MENG C L, ERMON S. Denoising diffusion implicit models[EB/OL]. [2020-10-06]. http://arxiv.org/abs/2010.02502.
[9]	SUN Z Y, LV T, YE S, et al. DiffPoseTalk: speech-driven stylistic 3D facial animation and head pose generation via diffusion models[J]. ACM Transactions on Graphics, 2024, 43(4): 1-9.
[10]	VAN DEN OORD A, VINYALS O, KAVUKCUOGLU K. Neural discrete representation learning[J]. Advances in Neural Information Processing Systems, 2017, 30.
[11]	CHU X G, GOSWAMI N, CUI Z T, et al. ARTalk: speech- driven 3D head animation via autoregressive model[C]// 2025 ACM SIGGRAPH Conference Papers. New York: ACM, 2025: 1-9.
[12]	MASSARO D W, COHEN M M, TABAIN M, et al. Animated speech: research progress and applications[M]//BAILLY G, PERRIER P, VATIKIOTIS-BATESON E. Audiovisual Speech Processing. Cambridge: Cambridge University Press, 2012: 309-345.
[13]	EDWARDS P, LANDRETH C, FIUME E, et al. JALI: an animator-centric viseme model for expressive lip synchronization[J]. ACM Transactions on Graphics, 2016, 35(4): 1-11.
[14]	BAEVSKI A, ZHOU Y H, MOHAMED A, et al. wav2vec 2.0: a framework for self-supervised learning of speech representations[J]. Advances in Neural Information Processing Systems, 2020, 33: 12449-12460.
[15]	HSU W N, BOLTE B, TSAI Y H H, et al. HuBERT: self-supervised speech representation learning by masked prediction of hidden units[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 3451-3460. DOI URL
[16]	PENG Z Q, WU H Y, SONG Z B, et al. EmoTalk: speech- driven emotional disentanglement for 3D face animation[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 20687-20697.
[17]	ZHANG C X, NI S F, FAN Z P, et al. 3D talking face with personalized pose dynamics[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 29(2): 1438-1449. DOI URL
[18]	CUDEIRO D, BOLKART T, LAIDLAW C, et al. Capture, learning, and synthesis of 3D speaking styles[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 10101-10111.
[19]	HAQUE K I, YUMAK Z. FaceXHuBERT: text-less speech-driven EXpressive 3D facial animation synthesis using self-supervised speech representation learning[C]// The 25th International Conference on Multimodal Interaction. New York: ACM, 2023: 282-291.
[20]	XING J B, XIA M H, ZHANG Y C, et al. CodeTalker: speech-driven 3D facial animation with discrete motion prior[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 12780-12790.
[21]	THAMBIRAJA B, HABIBIE I, ALIAKBARIAN S, et al. Imitator: personalized speech-driven 3D facial animation[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 20621-20631.
[22]	STAN S, HAQUE K I, YUMAK Z. FaceDiffuser: speech- driven 3D facial animation synthesis using diffusion[C]// The 16th ACM SIGGRAPH Conference on Motion, Interaction and Games. New York: ACM, 2023: 1-11.
[23]	QIAN S H, KIRSCHSTEIN T, SCHONEVELD L, et al. GaussianAvatars: photorealistic head avatars with rigged 3D Gaussians[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 20299-20309.
[24]	DANĚČEK R, CHHATRE K, TRIPATHI S, et al. Emotional speech-driven animation with content-emotion disentanglement[C]// 2023 SIGGRAPH Asia Conference Papers. New York: ACM, 2023: 1-13.
[25]	ZHANG Z M, LI L C, DING Y, et al. Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 3661-3670.