基于语义感知和混合物质点法的高斯动态重建

doi:10.11996/JG.j.2095-302X.2026040714

摘要/Abstract

摘要：

针对现有静态三维高斯泼溅(3DGS)技术缺乏物理动态真实性，重建的数字资产无法以符合物理规律的方式进行动态交互的问题，提出一种基于语义感知与混合物质点法(MPM)的高斯动态重建方法PhysGaussian-M2。该方法包含3个核心思路：首先，联合利用分割一切模型(SAM)与多模态大模型(LMM)，从视觉特征中自动推断物理本构参数，实现参数标定的自动化；其次，将三维高斯核同时作为可微渲染图元和MPM拉格朗日质点，构建物理与视觉统一的表征框架，避免传统流程中的数据格式失配；最后，在混合MPM框架下为不同语义区域分配独立本构模型，实现固体、液体等多相物质间的复杂物理交互。实验结果表明，语义驱动的参数估计模块能准确捕捉不同材质的物理规律，推断参数与标准值的对数误差控制在一个数量级以内；多相仿真实验成功模拟了刚体-流体耦合、弹性碰撞及颗粒坍塌等物理现象，生成场景具有高视觉保真度与物理可信度。该框架为创建逼真且可交互的物理数字孪生场景提供了端到端的技术路径。

关键词: 3D高斯泼溅, 物质点法, 物理感知重建, 语义分割, 多物理场模拟

Abstract:

Addressing the problem that existing static 3D Gaussian Splatting (3DGS) techniques lack physical dynamic realism and the reconstructed digital assets cannot perform dynamic interactions in a physically plausible manner, a Gaussian dynamic reconstruction method based on semantic perception and the hybrid Material Point Method (MPM), termed PhysGaussian-M2, was proposed. The proposed method encompassed three core ideas: First, the Segment Anything Model (SAM) and Large Multimodal Models (LMMs) were jointly utilized to automatically infer physical constitutive parameters from visual features, thereby enabling automated parameter calibration. Second, 3D Gaussian kernels were simultaneously employed as differentiable rendering primitives and MPM Lagrangian particles, establishing a unified physical-visual representation framework and avoiding data-format mismatches in traditional pipelines. Third, within the hybrid MPM framework, independent constitutive models were assigned to different semantic regions, enabling complex physical interactions among multi-phase materials such as solids and fluids. Experimental results demonstrated that the semantic-driven parameter-estimation module accurately captured the physical laws governing different materials, with the logarithmic error between inferred parameters and standard values controlled within one order of magnitude. Multi-phase simulation experiments successfully reproduced physical phenomena such as rigid-body-fluid coupling, elastic collisions, and granular collapse, and the generated scenes exhibited high visual fidelity and physical plausibility. The proposed framework provides an end-to-end technical pathway for creating realistic and interactive physical digital-twin scenes.

Key words: 3D Gaussian splatting, material point method, physics-aware reconstruction, semantic segmentation, multiphysics simulation

中图分类号:

TP391

许航, 谢雪光, 夏清, 高阳, 禹鹏, 胡珈皓. 基于语义感知和混合物质点法的高斯动态重建[J]. 图学学报, 2026, 47(4): 714-725.

XU Hang, XIE Xueguang, XIA Qing, GAO Yang, YU Peng, HU Jiahao. Gaussian dynamic reconstruction based on semantic perception and hybrid material point method[J]. Journal of Graphics, 2026, 47(4): 714-725.

图/表 8

参考文献 36

[1]	杨航, 陈瑞, 安仕鹏, 等. 深度学习背景下的图像三维重建技术进展综述[J]. 中国图象图形学报, 2023, 28(8): 2396-2409.
	YANG H, CHEN R, AN S P, et al. The growth of image-related three dimensional reconstruction techniques in deep learning-driven era: a critical summary[J]. Journal of Image and Graphics, 2023, 28(8): 2396-2409 (in Chinese). DOI URL
[2]	龙霄潇, 程新景, 朱昊, 等. 三维视觉前沿进展[J]. 中国图象图形学报, 2021, 26(6): 1389-1428.
	LONG X X, CHENG X J, ZHU H, et al. Recent progress in 3D vision[J]. Journal of Image and Graphics, 2021, 26(6): 1389-1428 (in Chinese). DOI URL
[3]	MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: representing scenes as neural radiance fields for view synthesis[J]. Communications of the ACM, 2022, 65(1): 99-106.
[4]	KERBL B, KOPANAS G, LEIMKÜHLER T, et al. 3D Gaussian splatting for real-time radiance field rendering[J]. ACM Transactions on Graphics, 2023, 42(4): 139.
[5]	BARRON J T, MILDENHALL B, VERBIN D, et al. Mip-NeRF 360: unbounded anti-aliased neural radiance fields[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 5470-5479.
[6]	MÜLLER T, EVANS A, SCHIED C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM Transactions on Graphics, 2022, 41(4): 102.
[7]	SCHÖNBERGER J L, FRAHM J M. Structure-from-motion revisited[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 4104-4113.
[8]	PUMAROLA A, CORONA E, PONS-MOLL G, et al. D-NeRF: neural radiance fields for dynamic scenes[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 10318-10327.
[9]	PARK K, SINHA U, BARRON J T, et al. Nerfies: deformable neural radiance fields[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 5865-5874.
[10]	WU G J, YI T R, FANG J M, et al. 4D Gaussian splatting for real-time dynamic scene rendering[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 16622-16632.
[11]	YUAN Y J, SUN Y T, LAI Y K, et al. NeRF-editing: geometry editing of neural radiance fields[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 18353-18364.
[12]	XU T H, HARADA T. Deforming radiance fields with cages[C]// The 17th European Conference on Computer Vision. Cham: Springer, 2022: 159-175.
[13]	QIAO Y L, GAO A, LIN M C. NeuPhysics: editable neural geometry and physics from monocular videos[C]// The 36th International Conference on Neural Information Processing Systems. New York: ACM, 2022: 933.
[14]	XIE T Y, ZONG Z S, QIU Y X, et al. PhysGaussian: physics-integrated 3D Gaussians for generative dynamics[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 4456-4467.
[15]	MÜLLER M, CHENTANEZ N, MACKLIN M. Simulating visual geometry[C]// The 9th International Conference on Motion in Games. New York: ACM, 2016: 31-38.
[16]	ZHAO H Y, WANG H, ZHAO X Y, et al. PhysSplat: efficient physics simulation for 3D scenes via MLLM-guided Gaussian splatting[C]// 2025 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2025. DOI: 10.1109/ICCV51701.2025.00498.
[17]	BONET J, WOOD R D. Nonlinear continuum mechanics for finite element analysis[M]. Cambridge: Cambridge University Press, 1997.
[18]	SIMO J C, HUGHES T J R. Computational inelasticity[M]. New York: Springer-Verlag, 1998: 241-278.
[19]	LI X, QIAO Y L, CHEN P Y, et al. PAC-NeRF: physics augmented continuum neural radiance fields for geometry-agnostic system identification[EB/OL]. [2025-01-15]. https://arxiv.org/abs/2303.05512.
[20]	CEN J Z, FANG J M, YANG C, et al. Segment any 3D Gaussians[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39(2): 1971-1979. DOI URL
[21]	YE M Q, DANELLJAN M, YU F S, et al. Gaussian grouping: segment and edit anything in 3D scenes[C]// The 18th European Conference on Computer Vision. Cham: Springer, 2025: 162-179.
[22]	YU Z, YE S, SUN Y L, et al. Deep learning method for predicting the mechanical properties of aluminum alloys with small data sets[J]. Materials Today Communications, 2021, 28: 102570. DOI URL
[23]	ZWICKER M, PFISTER H, VAN BAAR J, et al. EWA volume splatting[C]// 2001 Visualization. New York: IEEE Press, 2001: 29-538.
[24]	LIN Y T, DAI Z Z, ZHU S Y, et al. Gaussian-flow: 4D reconstruction with dynamic 3D Gaussian particle[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 21136-21145.
[25]	HU S, HONG F, XU L K, et al. GauHuman: articulated Gaussian Splatting from monocular human videos[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 20418-20431.
[26]	JIANG C F F, SCHROEDER C, TERAN J, et al. The material point method for simulating continuum materials[C]// ACM SIGGRAPH 2016 Courses. New York: ACM, 2016: 24.
[27]	SULSKY D, CHEN Z, SCHREYER H L. A particle method for history-dependent materials[J]. Computer Methods in Applied Mechanics and Engineering, 1994, 118(1/2): 179-196. DOI URL
[28]	JIANG C F F, GAST T, TERAN J. Anisotropic elastoplasticity for cloth, knit and hair frictional contact[J]. ACM Transactions on Graphics (TOG), 2017, 36(4): 152.
[29]	YUE Y H, SMITH B, BATTY C, et al. Continuum foam: a material point method for shear-dependent flows[J]. ACM Transactions on Graphics, 2015, 34(5): 160.
[30]	KLÁR G, GAST T, PRADHANA A, et al. Drucker-prager elastoplasticity for sand animation[J]. ACM Transactions on Graphics (TOG), 2016, 35(4): 103.
[31]	STOMAKHIN A, SCHROEDER C, CHAI L, et al. A material point method for snow simulation[J]. ACM Transactions on Graphics (TOG), 2013, 32(4): 102.
[32]	GAO M, WANG X L, WU K, et al. GPU optimization of material point methods[J]. ACM Transactions on Graphics, 2018, 37(6): 254.
[33]	HU Y M, LI T M, ANDERSON L, et al. Taichi: a language for high-performance computation on spatially sparse data structures[J]. ACM Transactions on Graphics, 2019, 38(6): 201.
[34]	LOPEZ-MARTIN M, LE CLAINCHE S, CARRO B. Model-free short-term fluid dynamics estimator with a deep 3D-convolutional neural network[J]. Expert Systems with Applications, 2021, 177: 114924. DOI URL
[35]	JIANG C, SCHROEDER C, SELLE A, et al. The affine particle-in-cell method[J]. ACM Transactions on Graphics, 2015, 34(4): 51.
[36]	QIN M H, LI W H, ZHOU J W, et al. LangSplat: 3D language Gaussian splatting[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 20051-20060.

实验材质	关键物理参数	标准真值	LMM 推断值	相对误差/%	参数属性
钢	杨氏模量($E$)	2.0×${{10}^{11}}$Pa	1.85×${{10}^{11}}$Pa	7.5	高刚性
橡胶	杨氏模量($E$)	1.0×${{10}^{7}}$Pa	1.25×${{10}^{7}}$Pa	25.0	超弹性
干沙	内摩擦角(${{\phi }_{f}}$)	35.0°	32.5°	7.1	颗粒摩擦
水	动力粘度($\eta $)	1.0×${{10}^{-3}}$P·as	1.15×${{10}^{-3}}$P·as	15.0	低粘度流体
凝胶	屈服应力(${{\tau }_{y}}$)	100 Pa	88 Pa	12.0	粘塑性

实验材质	关键物理参数	标准真值	LMM 推断值	相对误差/%	参数属性
钢	杨氏模量($E$)	2.0×${{10}^{11}}$Pa	1.85×${{10}^{11}}$Pa	7.5	高刚性
橡胶	杨氏模量($E$)	1.0×${{10}^{7}}$Pa	1.25×${{10}^{7}}$Pa	25.0	超弹性
干沙	内摩擦角(${{\phi }_{f}}$)	35.0°	32.5°	7.1	颗粒摩擦
水	动力粘度($\eta $)	1.0×${{10}^{-3}}$P·as	1.15×${{10}^{-3}}$P·as	15.0	低粘度流体
凝胶	屈服应力(${{\tau }_{y}}$)	100 Pa	88 Pa	12.0	粘塑性

实验模型	参数类型	材质推断	杨氏模量/Pa	泊松比	物理行为描述
Bunny	语义感知参数	金属	5.0×10⁵	0.33	刚性，碰撞后几乎不变形
Bunny	自定义对比参数	软体	8.0×10³	0.45	柔软，碰撞后压缩形变和弹跳
Armadillo	语义感知参数	软质黏土/沙土	1.0×10⁴	0.40	柔软，碰撞后压缩形变和弹跳
Armadillo	自定义对比参数	刚体	5.0×10⁵	0.33	刚性，碰撞后几乎不变形
Monkey	语义感知参数	橡胶	1.1×10⁴	0.48	超弹性，碰撞后形变并迅速回弹
Monkey	自定义对比参数	刚体	5.0×10⁵	0.33	刚性，碰撞后几乎不变形

实验模型	参数类型	材质推断	杨氏模量/Pa	泊松比	物理行为描述
Bunny	语义感知参数	金属	5.0×10⁵	0.33	刚性，碰撞后几乎不变形
Bunny	自定义对比参数	软体	8.0×10³	0.45	柔软，碰撞后压缩形变和弹跳
Armadillo	语义感知参数	软质黏土/沙土	1.0×10⁴	0.40	柔软，碰撞后压缩形变和弹跳
Armadillo	自定义对比参数	刚体	5.0×10⁵	0.33	刚性，碰撞后几乎不变形
Monkey	语义感知参数	橡胶	1.1×10⁴	0.48	超弹性，碰撞后形变并迅速回弹
Monkey	自定义对比参数	刚体	5.0×10⁵	0.33	刚性，碰撞后几乎不变形