Free sculpting system in virtual reality environment

doi:10.11996/JG.j.2095-302X.2025020345

Abstract

Abstract:

Shape modeling is a dynamic area in computer graphics, with virtual sculpting being one of the important paradigms in freeform shape modeling. Traditional virtual sculpting typically relies on desktop computers, where users manipulate meshes with controllers and view the models on two-dimensional displays. However, this approach poses issues such as limited viewing angles and poor immersion. With the emergence of virtual reality in recent years, the immersive interactive experience provides new possibilities for the implementation of virtual sculpting. By integrating virtual reality with virtual sculpting, a real-time virtual sculpting system was designed and implemented in a virtual reality environment based on quasi-uniform mesh as the structural foundation. The system design primarily encompasses surface point selection algorithms, mesh optimization techniques, mesh deformation strategies, and topology merging methods. Furthermore, a free topology algorithm was developed to provide higher degrees of freedom for sculpting modeling. For the general sculpting process, a series of user-friendly sculpting tools was implemented based on the aforementioned algorithms, ensuring that the mesh remains closed, manifold, and free of self-intersecting. Furthermore, to address the requirement for seamless fusion between arbitrary models, two fusion methods guided by signed distance fields were proposed, which utilized mesh deformation and mesh merging, respectively. The models created by the system can be applied to various scenarios. The experimental results validated the effectiveness, versatility, and user-friendliness of the proposed algorithms.

Key words: virtual reality, three-dimensional modeling, virtual sculpting, free topology, model merge

CLC Number:

TP391

ZHU Xiaoqiang, YANG Yifei. Free sculpting system in virtual reality environment[J]. Journal of Graphics, 2025, 46(2): 345-357.

Figures/Tables 24

Fig. 1 Paint brush form in VR environment ((a) Spherical brush; (b) Radial brush)

Fig. 2 The sculpting effects of different brush modes in different areas ((a) Dorsal area of the original dog model; (b) Neck area of the original dog model; (c) Sculpting the dorsal area with a spherical brush; (d) Sculpting the neck area with a spherical brush; (e) Sculpting the dorsal area with a radial brush; (f) Sculpting the neck area with a radial brush)

Fig. 3 Long edge division operation ((a) Structure before division; (b) Structure after division)

Fig. 4 Short edge folding operation ((a) Structure before folding; (b) Structure after folding)

Fig. 5 Optimization of the bunny model (genus=0) ((a) Original bunny model; (b) Optimized bunny model)

Fig. 6 Optimization of the cup model (genus?0) ((a) Original cup model; (b) Optimized cup model)

Fig. 7 Inflate and Deflate operations on the same sphere ((a) Inflate; (b) Deflate)

Fig. 8 Different operations on the same initial model ((a) Inflate; (b) Deflate; (c) Flatten; (d) Smooth)

Fig. 9 Topological fusion caused by Inflate operation ((a) Structure before fusion; (b) Structure after fusion)

Fig. 10 Merger of two non-adjacent vertices ((a) Structure before merger; (b) Structure after merger)

Fig. 11 Grid situation with or without step size limit ((a) Without step size limit; (b) With step size limit)

Fig. 12 Model generated by Free Brush based on grid deformation ((a) Original model; (b) Generated model)

Fig. 13 Result after moving points on the target model((a) Target model after point movement; (b) Partial area of the model to be integrated)

Fig. 14 Model generated by Free Brush based on grid fusion ((a) Original model; (b) Generated model)

Fig. 15 The handle interface ((a) The left-hand handle; (b) The right-hand handle)

Fig. 16 Mesh with or without automatic topology fusion mechanism ((a) Internal intersection within the model; (b) No internal intersections within the model; (c) The model grid with distortion; (d) The model grid without distortion; (e) Magnified view of the grid with distortion; (f) Magnified view of the grid without distortion)

Fig. 17 Mesh with and without position rationality detection ((a) Illegal operation; (b) Model with broken mesh; (c) Legal operation; (d) Model without broken mesh)

Table 1 Operation Time Consumption/ms

画刷模式	雕刻模式	选点	优化	变形	总时间
球形	Inflate	0.092	0.133	0.001 9	3.944
	Deflate	0.071	0.101	0.001 9	3.993
	Flatten	0.068	0.022	0.002 8	3.643
	Smooth	0.075	0.028	0.005 4	3.782
射线	Inflate	0.796	0.085	0.001 7	4.737
	Deflate	0.869	0.086	0.001 6	4.692
	Flatten	1.009	0.029	0.002 8	4.897
	Smooth	0.897	0.041	0.006 2	4.732

Table 2 User research results

雕刻系统	本文		Shapelab
雕刻系统	平均值	标准差	平均值	标准差
系统易用性	5.80	0.86	5.20	1.08
功能完整性	4.20	0.77	6.40	0.63
建模鲁棒性	6.33	0.49	6.47	0.52
细节实现度	5.60	0.63	6.20	0.56
拓扑自由性	5.73	0.46	5.67	0.72
整体评价	5.73	0.59	6.13	0.35

Fig. 18 Model created based on existing models ((a) Original model; (b) Sculpted model)

Fig. 19 Colored lantern model

Fig. 20 User-created butterfly-like creature model

Fig. 21 New model obtained by blending two models using freehand brush

Fig. 22 Vase sculpting process ((a) Create the basic shape of vase; (b) Add handles; (c) Create suspended ring; (d) Add overall decoration)

References 32

[1]	林莹莹, 蔡睿凡, 朱雨真, 等. 基于Leap Motion的虚拟现实陶艺体验系统[J]. 图学学报, 2020, 41(1): 57-65. DOI
	LIN Y Y, CAI R F, ZHU Y Z, et al. Virtual reality pottery modeling system based on leap motion[J]. Journal of Graphics, 2020, 41(1): 57-65 (in Chinese).
[2]	王浩淼, 桑胜举, 段晓东, 等. 虚拟现实环境下的协同式三维建模方法[J]. 图学学报, 2024, 45(1): 169-182. DOI
	WANG H M, SANG S J, DUAN X D, et al. Collaborative 3D modeling technique in virtual reality[J]. Journal of Graphics, 2024, 45(1): 169-182 (in Chinese). DOI
[3]	GEIGER A, ZIEGLER J, STILLER C. StereoScan: dense 3D reconstruction in real-time[C]// 2011 IEEE Intelligent Vehicles Symposium. New York: IEEE Press, 2011: 963-968.
[4]	IZADI S, KIM D, HILLIGES O, et al. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera[C]// The 24th Annual ACM Symposium on User Interface Software and Technology. New York: ACM, 2011: 559-568.
[5]	MOURAGNON E, LHUILLIER M, DHOME M, et al. Real time localization and 3D reconstruction[C]// 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2006: 363-370.
[6]	DAVIES R H, COOTES T F, TWINING C J, et al. An information theoretic approach to statistical shape modelling[EB/OL]. [2024-06-27]. https://personalpages.manchester.ac.uk/staff/timothy.f.cootes/Papers/bmvc01_davies.pdf.
[7]	WANG H X, MARKOSIAN L. Free-form sketch[C]// The 4th Eurographics Workshop on Sketch-based Interfaces and Modeling. New York: ACM, 2007: 53-58.
[8]	YU E, ARORA R, BÆRENTZEN J A, et al. Piecewise-smooth surface fitting onto unstructured 3D sketches[J]. ACM Transactions on Graphics, 2022, 41(4): 88.
[9]	MIKAEILI A, PEREL O, SAFAEE M, et al. SKED: sketch-guided text-based 3D editing[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 14561-14573.
[10]	LAMOUSIN H J, WAGGENSPACK N N. NURBS-based free-form deformations[J]. IEEE Computer Graphics and Applications, 1994, 14(6): 59-65.
[11]	WONG J P Y, LAU R W H, MA L Z. Virtual 3D sculpting[J]. The Journal of Visualization and Computer Animation, 2000, 11(3): 155-166.
[12]	MCDONNELL K T, QIN H, WLODARCZYK R A. Virtual clay: a real-time sculpting system with haptic toolkits[C]// The 2001 Symposium on Interactive 3D Graphics. New York: ACM, 2001: 179-190.
[13]	KEEFE D F, FELIZ D A, MOSCOVICH T, et al. CavePainting: a fully immersive 3D artistic medium and interactive experience[C]// The 2001 Symposium on Interactive 3D Graphics. New York: ACM, 2001: 85-93.
[14]	GALYEAN T A, HUGHES J F. Sculpting: an interactive volumetric modeling technique[J]. ACM SIGGRAPH Computer Graphics, 1991, 25(4): 267-274.
[15]	LORENSEN W E, CLINE H E. Marching cubes: a high resolution 3D surface construction algorithm[J]. ACM SIGGRAPH Computer Graphics, 1987, 21(4): 163-169.
[16]	STÃNCULESCU L, CHAINE R, CANI M P. Freestyle: sculpting meshes with self-adaptive topology[J]. Computers & Graphics, 2011, 35(3): 614-622.
[17]	JANG S A, KIM H I, WOO W, et al. AiRSculpt: a wearable augmented reality 3D sculpting system[C]// The 2nd International Conference on Distributed, Ambient, and Pervasive Interactions. Cham: Springer, 2014: 130-141.
[18]	LU P, SHENG B, LUO S M, et al. Image-based non-photorealistic rendering for realtime virtual sculpting[J]. Multimedia Tools and Applications, 2015, 74(21): 9697-9714.
[19]	CALLENS E, DANIEAU F, COSTES A, et al. A tangible surface for digital sculpting in virtual environments[C]// The 11th International Conference on Haptics:Science, Technology, and Applications. Cham: Springer, 2018: 157-168.
[20]	MILLIEZ A, WAND M, CANI M, et al. Mutable elastic models for sculpting structured shapes[J]. Computer Graphics Forum, 2013, 32(2pt1): 21-30.
[21]	GAO Z H, LI J W, WANG H Q, et al. DigiClay: an interactive installation for virtual pottery using motion sensing technology[C]// The 4th International Conference on Virtual Reality. New York: ACM, 2018: 126-132.
[22]	朱晓强, 余涛. 头戴设备VR环境下基于网格变形的交互雕刻建模[J]. 浙江大学学报(工学版), 2018, 52(3): 599-604.
	ZHU X Q, YU T. Interactive sculpture modeling based on mesh deformation in HMD VR environment[J]. Journal of Zhejiang University (Engineering Science), 2018, 52(3): 599-604 (in Chinese).
[23]	PARKER S G, BIGLER J, DIETRICH A, et al. Optix: a general purpose ray tracing engine[J]. ACM Transactions on Graphics, 2010, 29(4): 1-13.
[24]	FERLEY E, CANI M P, GASCUEL J D. Practical volumetric sculpting[J]. The Visual Computer, 2000, 16(8): 469-480.
[25]	FERLEY E, CANI M P, GASCUEL J D. Resolution adaptive volume sculpting[J]. Graphical Models, 2001, 63(6): 459-478.
[26]	CHEN C W, HU M C, CHU W T, et al. A real-time sculpting and terrain generation system for interactive content creation[J]. IEEE Access, 2021, 9: 114914-114928.
[27]	BRESENHAM J E. Algorithm for computer control of a digital plotter[J]. IBM Systems Journal, 1965, 4(1): 25-30.
[28]	GAO Z H, WANG H Q, FENG G S, et al. RealPot: an immersive virtual pottery system with handheld haptic devices[J]. Multimedia Tools and Applications, 2019, 78(18): 26569-26596.
[29]	DASHTI S, PRAKASH E, NAVARRO-NEWBALL A A, et al. PotteryVR: virtual reality pottery[J]. The Visual Computer, 2022, 38(12): 4035-4055.
[30]	POOLE B, JAIN A, BARRON J T, et al. DreamFusion:text-to-3D using 2D diffusion[EB/OL]. [2024-10-15]. https://arxiv.org/abs/2209.14988.
[31]	JAMBON C, KERBL B, KOPANAS G, et al. NeRFshop: interactive editing of neural radiance fields[J]. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2023, 6(1): 1.
[32]	DING L H, DONG S C, HUANG Z P, et al. Text-to-3D generation with bidirectional diffusion using both 2D and 3D priors[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 5115-5124.