Gaussian dynamic reconstruction based on semantic perception and hybrid material point method

doi:10.11996/JG.j.2095-302X.2026040714

Abstract

Abstract:

Addressing the problem that existing static 3D Gaussian Splatting (3DGS) techniques lack physical dynamic realism and the reconstructed digital assets cannot perform dynamic interactions in a physically plausible manner, a Gaussian dynamic reconstruction method based on semantic perception and the hybrid Material Point Method (MPM), termed PhysGaussian-M2, was proposed. The proposed method encompassed three core ideas: First, the Segment Anything Model (SAM) and Large Multimodal Models (LMMs) were jointly utilized to automatically infer physical constitutive parameters from visual features, thereby enabling automated parameter calibration. Second, 3D Gaussian kernels were simultaneously employed as differentiable rendering primitives and MPM Lagrangian particles, establishing a unified physical-visual representation framework and avoiding data-format mismatches in traditional pipelines. Third, within the hybrid MPM framework, independent constitutive models were assigned to different semantic regions, enabling complex physical interactions among multi-phase materials such as solids and fluids. Experimental results demonstrated that the semantic-driven parameter-estimation module accurately captured the physical laws governing different materials, with the logarithmic error between inferred parameters and standard values controlled within one order of magnitude. Multi-phase simulation experiments successfully reproduced physical phenomena such as rigid-body-fluid coupling, elastic collisions, and granular collapse, and the generated scenes exhibited high visual fidelity and physical plausibility. The proposed framework provides an end-to-end technical pathway for creating realistic and interactive physical digital-twin scenes.

Key words: 3D Gaussian splatting, material point method, physics-aware reconstruction, semantic segmentation, multiphysics simulation

CLC Number:

TP391

XU Hang, XIE Xueguang, XIA Qing, GAO Yang, YU Peng, HU Jiahao. Gaussian dynamic reconstruction based on semantic perception and hybrid material point method[J]. Journal of Graphics, 2026, 47(4): 714-725.

Figures/Tables 8

References 36

[1]	杨航, 陈瑞, 安仕鹏, 等. 深度学习背景下的图像三维重建技术进展综述[J]. 中国图象图形学报, 2023, 28(8): 2396-2409.
	YANG H, CHEN R, AN S P, et al. The growth of image-related three dimensional reconstruction techniques in deep learning-driven era: a critical summary[J]. Journal of Image and Graphics, 2023, 28(8): 2396-2409 (in Chinese). DOI URL
[2]	龙霄潇, 程新景, 朱昊, 等. 三维视觉前沿进展[J]. 中国图象图形学报, 2021, 26(6): 1389-1428.
	LONG X X, CHENG X J, ZHU H, et al. Recent progress in 3D vision[J]. Journal of Image and Graphics, 2021, 26(6): 1389-1428 (in Chinese). DOI URL
[3]	MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2022, 65(1): 99-106.
[4]	KERBL B, KOPANAS G, LEIMKÜHLER T, et al. 3D Gaussian splatting for real-time radiance field rendering[J]. ACM Transactions on Graphics, 2023, 42(4): 139.
[5]	BARRON J T, MILDENHALL B, VERBIN D, et al. Mip-NeRF 360: unbounded anti-aliased neural radiance fields[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 5470-5479.
[6]	MÜLLER T, EVANS A, SCHIED C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM Transactions on Graphics, 2022, 41(4): 102.
[7]	SCHÖNBERGER J L, FRAHM J M. Structure-from-motion revisited[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 4104-4113.
[8]	PUMAROLA A, CORONA E, PONS-MOLL G, et al. D-NeRF: neural radiance fields for dynamic scenes[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 10318-10327.
[9]	PARK K, SINHA U, BARRON J T, et al. Nerfies: deformable neural radiance fields[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 5865-5874.
[10]	WU G J, YI T R, FANG J M, et al. 4D Gaussian splatting for real-time dynamic scene rendering[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 16622-16632.
[11]	YUAN Y J, SUN Y T, LAI Y K, et al. NeRF-editing: geometry editing of neural radiance fields[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 18353-18364.
[12]	XU T H, HARADA T. Deforming radiance fields with cages[C]// The 17th European Conference on Computer Vision. Cham: Springer, 2022: 159-175.
[13]	QIAO Y L, GAO A, LIN M C. NeuPhysics: editable neural geometry and physics from monocular videos[C]// The 36th International Conference on Neural Information Processing Systems. New York: ACM, 2022: 933.
[14]	XIE T Y, ZONG Z S, QIU Y X, et al. PhysGaussian: physics-integrated 3D Gaussians for generative dynamics[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 4456-4467.
[15]	MÜLLER M, CHENTANEZ N, MACKLIN M. Simulating visual geometry[C]// The 9th International Conference on Motion in Games. New York: ACM, 2016: 31-38.
[16]	ZHAO H Y, WANG H, ZHAO X Y, et al. PhysSplat: efficient physics simulation for 3D scenes via MLLM-guided Gaussian splatting[C]// 2025 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2025. DOI: 10.1109/ICCV51701.2025.00498.
[17]	BONET J, WOOD R D. Nonlinear continuum mechanics for finite element analysis[M]. Cambridge: Cambridge University Press, 1997.
[18]	SIMO J C, HUGHES T J R. Computational inelasticity[M]. New York: Springer-Verlag, 1998: 241-278.
[19]	LI X, QIAO Y L, CHEN P Y, et al. PAC-NeRF: physics augmented continuum neural radiance fields for geometry-agnostic system identification[EB/OL]. [2025-01-15]. https://arxiv.org/abs/2303.05512.
[20]	CEN J Z, FANG J M, YANG C, et al. Segment any 3D Gaussians[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39(2): 1971-1979. DOI URL
[21]	YE M Q, DANELLJAN M, YU F S, et al. Gaussian grouping: segment and edit anything in 3D scenes[C]// The 18th European Conference on Computer Vision. Cham: Springer, 2025: 162-179.
[22]	YU Z, YE S, SUN Y L, et al. Deep learning method for predicting the mechanical properties of aluminum alloys with small data sets[J]. Materials Today Communications, 2021, 28: 102570. DOI URL
[23]	ZWICKER M, PFISTER H, VAN BAAR J, et al. EWA volume splatting[C]// 2001 Visualization. New York: IEEE Press, 2001: 29-538.
[24]	LIN Y T, DAI Z Z, ZHU S Y, et al. Gaussian-flow: 4D reconstruction with dynamic 3D Gaussian particle[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 21136-21145.
[25]	HU S, HONG F, XU L K, et al. GauHuman: articulated Gaussian Splatting from monocular human videos[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 20418-20431.
[26]	JIANG C F F, SCHROEDER C, TERAN J, et al. The material point method for simulating continuum materials[C]// ACM SIGGRAPH 2016 Courses. New York: ACM, 2016: 24.
[27]	SULSKY D, CHEN Z, SCHREYER H L. A particle method for history-dependent materials[J]. Computer Methods in Applied Mechanics and Engineering, 1994, 118(1/2): 179-196. DOI URL
[28]	JIANG C F F, GAST T, TERAN J. Anisotropic elastoplasticity for cloth, knit and hair frictional contact[J]. ACM Transactions on Graphics (TOG), 2017, 36(4): 152.
[29]	YUE Y H, SMITH B, BATTY C, et al. Continuum foam: a material point method for shear-dependent flows[J]. ACM Transactions on Graphics, 2015, 34(5): 160.
[30]	KLÁR G, GAST T, PRADHANA A, et al. Drucker-prager elastoplasticity for sand animation[J]. ACM Transactions on Graphics (TOG), 2016, 35(4): 103.
[31]	STOMAKHIN A, SCHROEDER C, CHAI L, et al. A material point method for snow simulation[J]. ACM Transactions on Graphics (TOG), 2013, 32(4): 102.
[32]	GAO M, WANG X L, WU K, et al. GPU optimization of material point methods[J]. ACM Transactions on Graphics, 2018, 37(6): 254.
[33]	HU Y M, LI T M, ANDERSON L, et al. Taichi: a language for high-performance computation on spatially sparse data structures[J]. ACM Transactions on Graphics, 2019, 38(6): 201.
[34]	LOPEZ-MARTIN M, LE CLAINCHE S, CARRO B. Model-free short-term fluid dynamics estimator with a deep 3D-convolutional neural network[J]. Expert Systems with Applications, 2021, 177: 114924. DOI URL
[35]	JIANG C, SCHROEDER C, SELLE A, et al. The affine particle-in-cell method[J]. ACM Transactions on Graphics, 2015, 34(4): 51.
[36]	QIN M H, LI W H, ZHOU J W, et al. LangSplat: 3D language Gaussian splatting[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 20051-20060.

实验材质	关键物理参数	标准真值	LMM 推断值	相对误差/%	参数属性
钢	杨氏模量($E$)	2.0×${{10}^{11}}$Pa	1.85×${{10}^{11}}$Pa	7.5	高刚性
橡胶	杨氏模量($E$)	1.0×${{10}^{7}}$Pa	1.25×${{10}^{7}}$Pa	25.0	超弹性
干沙	内摩擦角(${{\phi }_{f}}$)	35.0°	32.5°	7.1	颗粒摩擦
水	动力粘度($\eta $)	1.0×${{10}^{-3}}$P·as	1.15×${{10}^{-3}}$P·as	15.0	低粘度流体
凝胶	屈服应力(${{\tau }_{y}}$)	100 Pa	88 Pa	12.0	粘塑性

实验材质	关键物理参数	标准真值	LMM 推断值	相对误差/%	参数属性
钢	杨氏模量($E$)	2.0×${{10}^{11}}$Pa	1.85×${{10}^{11}}$Pa	7.5	高刚性
橡胶	杨氏模量($E$)	1.0×${{10}^{7}}$Pa	1.25×${{10}^{7}}$Pa	25.0	超弹性
干沙	内摩擦角(${{\phi }_{f}}$)	35.0°	32.5°	7.1	颗粒摩擦
水	动力粘度($\eta $)	1.0×${{10}^{-3}}$P·as	1.15×${{10}^{-3}}$P·as	15.0	低粘度流体
凝胶	屈服应力(${{\tau }_{y}}$)	100 Pa	88 Pa	12.0	粘塑性

实验模型	参数类型	材质推断	杨氏模量/Pa	泊松比	物理行为描述
Bunny	语义感知参数	金属	5.0×10⁵	0.33	刚性，碰撞后几乎不变形
Bunny	自定义对比参数	软体	8.0×10³	0.45	柔软，碰撞后压缩形变和弹跳
Armadillo	语义感知参数	软质黏土/沙土	1.0×10⁴	0.40	柔软，碰撞后压缩形变和弹跳
Armadillo	自定义对比参数	刚体	5.0×10⁵	0.33	刚性，碰撞后几乎不变形
Monkey	语义感知参数	橡胶	1.1×10⁴	0.48	超弹性，碰撞后形变并迅速回弹
Monkey	自定义对比参数	刚体	5.0×10⁵	0.33	刚性，碰撞后几乎不变形

实验模型	参数类型	材质推断	杨氏模量/Pa	泊松比	物理行为描述
Bunny	语义感知参数	金属	5.0×10⁵	0.33	刚性，碰撞后几乎不变形
Bunny	自定义对比参数	软体	8.0×10³	0.45	柔软，碰撞后压缩形变和弹跳
Armadillo	语义感知参数	软质黏土/沙土	1.0×10⁴	0.40	柔软，碰撞后压缩形变和弹跳
Armadillo	自定义对比参数	刚体	5.0×10⁵	0.33	刚性，碰撞后几乎不变形
Monkey	语义感知参数	橡胶	1.1×10⁴	0.48	超弹性，碰撞后形变并迅速回弹
Monkey	自定义对比参数	刚体	5.0×10⁵	0.33	刚性，碰撞后几乎不变形