Monocular 3D face reconstruction based on inverse rendering networks

doi:10.11996/JG.j.2095-302X.2026051025

Abstract

Abstract:

To balance inference efficiency, animation control, and relighting realism in monocular 3D face reconstruction, a feed-forward inverse-rendering framework was proposed. 3D Gaussian Splatting (3DGS) was combined with the FLAME prior, and Physically Based Rendering (PBR) attributes were introduced into the Gaussian primitives. A shared image encoder jointly predicts 3DGS features and an HDR illumination map, unifying the inference of geometry, materials, and lighting from a single image. To alleviate material-lighting ambiguity, IC-Light and Sapiens are introduced for screen-space pseudo-supervision of albedo and normal. An animation-aware normal updating mechanism and temporal consistency constraints further stabilize training. On the VFHQ dataset, the method achieved 22.34 dB PSNR and 0.855 SSIM for self-reenactment, outperforming the LAM baseline by 0.93 dB PSNR while maintaining sub-10 ms per-frame latency. Comparisons under the VFHQ-Test protocol showed significant advantages in reconstruction fidelity, driving accuracy, and novel-illumination consistency. The reconstructed PBR-3DGS representation supported natural FLAME-driven animation while preserving rigorous specular highlights and smooth shading under novel lighting, effectively balancing quality, efficiency, and physical consistency.

Key words: 3D face reconstruction, inverse rendering, 3D Gaussian splatting, relighting, digital human

CLC Number:

TP 391.41

LI Guohao, LV Chengxin, YANG Hongyu. Monocular 3D face reconstruction based on inverse rendering networks[J]. Journal of Graphics, 2026, 47(5): 1025-1032.

Figures/Tables 8

References 31

[1]	BLANZ V, VETTER T. A morphable model for the synthesis of 3D faces[C]// The 26th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1999: 187-194.
[2]	BLANZ V, VETTER T. Face recognition based on fitting a 3D morphable model[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(9): 1063-1074. DOI URL
[3]	TEWARI A, ZOLLHÖFER M, KIM H, et al. MoFA: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 3735-3744.
[4]	DENG Y, YANG J L, XU S C, et al. Accurate 3D face reconstruction with weakly-supervised learning: from single image to image set[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2019: 285-295.
[5]	FENG Y, FENG H W, BLACK M J, et al. Learning an animatable detailed 3D face model from in-the-wild images[J]. ACM Transactions on Graphics, 2021, 40(4): 88.
[6]	GECER B, PLOUMPIS S, KOTSIA I, et al. GANFIT: generative adversarial network fitting for high fidelity 3D face reconstruction[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 1155-1164.
[7]	MALLIKARJUN B R, TEWARI A, OH T H, et al. Monocular reconstruction of neural face reflectance fields[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 4789-4798.
[8]	KERBL B, KOPANAS G, LEIMKÜHLER T, et al. 3D Gaussian splatting for real-time radiance field rendering[J]. ACM Transactions on Graphics, 2023, 42(4): 139.
[9]	MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: representing scenes as neural radiance fields for view synthesis[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 405-421.
[10]	ZHENG Y F, ABREVAYA V F, BÜHLER M C, et al. I M Avatar: implicit morphable head avatars from videos[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 13535-13545.
[11]	MA S J, WENG Y L, SHAO T J, et al. 3D Gaussian blendshapes for head avatar animation[C]// ACM SIGGRAPH 2024 Conference Papers. New York: ACM, 2024: 60.
[12]	HE Y S, GU X D, YE X D, et al. LAM: large avatar model for one-shot animatable Gaussian head[C]// Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers. New York: ACM, 2025: 27.
[13]	CHU X, HARADA T. Generalizable and animatable Gaussian head avatar[C]// Advances in Neural Information Processing Systems 37. Red Hook: Curran Associates, Inc., 2024: 57642-57670.
[14]	ZHANG L, RAO A, AGRAWALA M. Scaling in-the-wild training for diffusion-based illumination harmonization and editing by imposing consistent light transport[EB/OL]. [2026-01-11]. https://openreview.net/forum?id=u1CQYxRI1H.
[15]	BHARADWAJ S, ZHENG Y F, HILLIGES O, et al. FLARE: fast learning of animatable and relightable mesh avatars[J]. ACM Transactions on Graphics, 2023, 42(6): 204.
[16]	SAITO S, SCHWARTZ G, SIMON T, et al. Relightable Gaussian codec avatars[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 130-141.
[17]	LI M X, CAO C, SCHWARTZ G, et al. URAvatar: universal relightable Gaussian codec avatars[C]// 2024 SIGGRAPH Asia Conference Papers. New York: ACM, 2024: 128.
[18]	ZHAO Z Y, LIU C J, CHEN H Y, et al. Total-Editing: head avatar with editable appearance, motion, and lighting[C]// 2025 IEEE/CVF International Conference on Computer Vision Workshops. New York: IEEE Press, 2025: 2622-2633.
[19]	GAFNI G, THIES J, ZOLLHÖFER M, et al. Dynamic neural radiance fields for monocular 4D facial avatar reconstruction[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 8645-8654.
[20]	HONG Y F, PENG B, XIAO H Y, et al. HeadNeRF: a realtime NeRF-based parametric head model[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 20342-20352.
[21]	GRASSAL P W, PRINZLER M, LEISTNER T, et al. Neural head avatars from monocular RGB videos[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 18632-18643.
[22]	ZIELONKA W, BOLKART T, THIES J. Instant volumetric head avatars[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 4574-4584.
[23]	ZHENG Y F, WANG Y F, WETZSTEIN G, et al. PointAvatar: deformable point-based head avatars from videos[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 21057-21067.
[24]	RAO P, FOX G, MEKA A, et al. Lite2Relight: 3D-aware single image portrait relighting[C]// 2024 ACM SIGGRAPH Conference Papers. New York: ACM, 2024: 41.
[25]	MA H Y, ZHANG T, SUN S L, et al. CVTHead: one-shot controllable head avatar with vertex-feature transformer[C]// 2024 IEEE/CVF Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2024: 6119-6129.
[26]	QIAN S H, KIRSCHSTEIN T, SCHONEVELD L, et al. GaussianAvatars: photorealistic head avatars with rigged 3D Gaussians[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 20299-20309.
[27]	XIANG J, GAO X, GUO Y D, et al. FlashAvatar: high-fidelity head avatar with efficient Gaussian embedding[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 1802-1812.
[28]	CHEN Y F, WANG L Z, LI Q J, et al. MonoGaussianAvatar: monocular Gaussian point-based head avatar[C]// 2024 ACM SIGGRAPH Conference Papers. New York: ACM, 2024: 58.
[29]	OQUAB M, DARCET T, MOUTAKANNI T, et al. DINOv2: learning robust visual features without supervision[J]. Transactions on Machine Learning Research, 2024, 2024: 1-32.
[30]	GAO J, GU C, LIN Y T, et al. Relightable 3D Gaussians:realistic point cloud relighting with BRDF decomposition and ray tracing[C]// The 18th European Conference on Computer Vision. Cham: Springer, 2024: 73-89.
[31]	KHIRODKAR R, BAGAUTDINOV T, MARTINEZ J, et al. Sapiens: foundation for human vision models[C]// The 18th European Conference on Computer Vision. Cham: Springer, 2024: 206-228.

方法	重建时间/s	单帧驱动渲染/ms	支持重光照	表示形式
GAGAvatar	1.0	15	否	3DGS+超分辨
LAM-20K	1.3	8	否	3DGS
本文方法	1.5	9	是	PBR-3DGS

方法	重建时间/s	单帧驱动渲染/ms	支持重光照	表示形式
GAGAvatar	1.0	15	否	3DGS+超分辨
LAM-20K	1.3	8	否	3DGS
本文方法	1.5	9	是	PBR-3DGS

方法	PSNR/dB ↑	SSIM↑	LPIPS↓	AKD↓	AED↓
CVTHead	18.43	0.706	0.317	5.678	0.261
Total-Editing	20.30	0.730	0.226	—	0.148
GPAvatar	21.04	0.807	0.150	4.226	0.255
GAGAvatar	21.83	0.818	0.122	3.349	0.253
LAM-20K	21.41	0.843	0.123	2.154	0.272
本文方法	22.34	0.855	0.144	2.141	0.259

方法	PSNR/dB ↑	SSIM↑	LPIPS↓	AKD↓	AED↓
CVTHead	18.43	0.706	0.317	5.678	0.261
Total-Editing	20.30	0.730	0.226	—	0.148
GPAvatar	21.04	0.807	0.150	4.226	0.255
GAGAvatar	21.83	0.818	0.122	3.349	0.253
LAM-20K	21.41	0.843	0.123	2.154	0.272
本文方法	22.34	0.855	0.144	2.141	0.259

设置	PSNR /dB	SSIM	LPIPS	自身份AED	AKD	跨身份AED
基线方法	21.66	0.850	0.143	0.139	2.051	0.269
+反照率伪监督	21.80	0.851	0.148	0.136	2.084	0.263
+法向量伪监督	21.90	0.853	0.135	0.128	2.083	0.270
+双伪监督	21.94	0.853	0.139	0.133	2.079	0.265
+IBL+伪监督	22.34	0.855	0.144	0.144	2.141	0.259