Hand reconstruction incorporating biomechanical constraints and multi-modal data

doi:10.11996/JG.j.2095-302X.2023040794

Abstract

Abstract:

To address the high cost and slow response of current monocular hand reconstruction, a method for hand 3D reconstruction using a monocular camera to acquire hand shape and posture was proposed. The method adopted a deep learning-based architecture that used image data with 2D and 3D annotations, as well as hand motion capture data for training. Firstly, 3D joint positions were accurately regressed and mapped to joint rotations through a joint detection module (3DHandNet) and an inverse kinematic module (IRNet). Then, biomechanical constraints were introduced to achieve high-quality mesh image alignment for real-time predictions. Finally, the resulting prediction vector with joint rotation representation was input to the hand mesh template to fit the hand shape. This approach was more suitable for computer vision and graphics applications compared to only regressing 3D joint positions. Experimental results on a benchmark dataset demonstrated that the proposed method achieves real-time runtime performance (60 fps) and high reconstruction accuracy, outperforming current methods in terms of hand posture estimation accuracy and hand image alignment.

Key words: 3D reconstruction of the hand, biomechanical constraints, multi-modal data, hand joint detection module, inverse kinematics module

CLC Number:

TP391

XUE Hao-wei, WANG Mei-li. Hand reconstruction incorporating biomechanical constraints and multi-modal data[J]. Journal of Graphics, 2023, 44(4): 794-800.

Figures/Tables 10

Fig. 1 Display of 3D hand reconstruction results ((a) Input image 1; (b) Output result 1; (c) Mesh overlay 1; (d) Input image 2; (e) Output result 2; (f) Mesh overlay 2)

Fig. 2 Overview of our architecture

Fig. 3 Visualization of the constructed 3DHandData dataset ((a) All angles; (b) Real 3D joint position)

Fig. 4 Description of biomechajiegounical structure

Fig. 5 Show our results for several challenging scenarios ((a) Input image; (b) Output result; (c) Mesh overlay)

Fig. 6 Shape and alignment effects of 3D reconstruction ((a) Input image; (b) 2D pose; (c) 3D pose; (d) Output result; (e) Mesh overlay)

Fig. 7 Comparison with the methods in literature [3] and [15] ((a) Input image; (b) Literature [15]; (c) Literature [3]; (d) Ours)

Table 1 Comparisons were made with state-of-the-art methods on four public datasets

Method	AUC of PCK
Method	DO	ED	STB	RHD
本文方法	0.953	0.823	0.896	0.950(*)
文献[7]	0.948	0.811	0.898(*)	0.856(*)
文献[3]	-	-	0.998(*)	0.920(*)
文献[4]	0.825	-	0.995(*)	0.901(*)
文献[17]	-	-	0.996(*)	0.943(*)
文献[11]	0.650	-	0.995(*)	0.926(*)
文献[8]	0.912	-	0.994(*)	-
文献[16]	0.763	0.674	0.994(*)	-
文献[5]	0.672	0.543	0.994(*)	-
文献[19]	0.573	-	0.948(*)	0.670(*)

Table 2 Ablation study

序号	Variants of our method	AUC of PCK
序号	Variants of our method	DO	ED	RHD
1	Full	0.953±0	0.823±0	0.950±0
2	w/o IRNet	0.913±0	0.802±0.04	0.808±0.07
3	w/o BMC	0.930±5e⁻⁴	0.752±5e⁻⁴	0.925±0.04
4	w/o MSSGesture	0.926±0.05	0.823±0.10	0.809±0.11
5	w/o 3DHandData	0.873±0	0.712±0.02	0.738±0.04

Fig. 8 As a baseline, was compared with the literature [4]

References 19

[1]	童立靖, 李嘉伟. 一种基于改进PointNet++网络的三维手姿估计方法[J]. 图学学报, 2022, 43(5): 892-900.
	TONG L J, LI J W. A 3D hand pose estimation method based on improved PointNet++[J]. Journal of Graphics, 2022, 43(5): 892-900 (in Chinese).
[2]	TANG X, WANG T Y, FU C W. Towards accurate alignment in real-time 3D hand-mesh reconstruction[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 11698-11707.
[3]	GE L H, REN Z, LI Y C, et al. 3D hand shape and pose estimation from a single RGB image[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 10825-10834.
[4]	ZHANG X, LI Q, MO H, et al. End-to-end hand mesh recovery from a monocular RGB image[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2020: 2354-2364.
[5]	IQBAL U, MOLCHANOV P, BREUEL T, et al. Hand pose estimation via latent 2.5D heatmap regression[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 125-143.
[6]	CAI Y J, GE L H, CAI J F, et al. Weakly-supervised 3D hand pose estimation from monocular RGB images[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 678-694.
[7]	ROMERO J, TZIONAS D, BLACK M J. Embodied hands: modeling and capturing hands and bodies together[J]. ACM Transactions on Graphics, 2017, 36(6): 245. 1-245.17.
[8]	ZHOU Y X, HABERMANN M, XU W P, et al. Monocular real-time hand shape and motion capture using multi-modal data[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 5345-5354.
[9]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 770-778.
[10]	ARMAGAN A, GARCIA-HERNANDO G, BAEK S, et al. Measuringgeneralisation to unseen viewpoints, articulations, shapes and objects for 3D hand pose estimation under hand-object interaction[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 85-101.
[11]	BAEK S, KIM K I, KIM T K. Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1067-1076.
[12]	ALBRECHT I, HABER J, SEIDEL H P. Construction and animation of anatomically based human hand models[C]// 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Cham: Springer International Publishing, 2003: 98-109.
[13]	SPURR A, IQBAL U, MOLCHANOV P, et al. Weakly supervised 3D hand pose estimation via biomechanical constraints[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 211-228.
[14]	范晶晶, 薛皓玮, 吴欣鸿, 等. 引入重影特征映射和通道注意力机制的手势识别算法[J]. 计算机辅助设计与图形学学报, 2022, 34(3): 403-414.
	FAN J J, XUE H W, WU X H, et al. Gesture recognition algorithm introducing ghost feature mapping and channel attention mechanism[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(3): 403-414 (in Chinese).
[15]	HASSON Y, VAROL G, TZIONAS D, et al. Learning joint reconstruction of hands and manipulated objects[EB/OL]. [2022-06-15]. https://arxiv.org/abs/1904.05767.
[16]	ZIMMERMANN C, BROX T. Learning to estimate 3D hand pose from single RGB images[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 4913-4921.
[17]	ZHANG J, JIAO J, CHEN M, et al. 3D hand pose tracking and estimation using stereo matching[EB/OL]. [2022-06-15]. https://arxiv.org/abs/1610.07214v1.
[18]	SRIDHAR S, MUELLER F, ZOLLHÖFER M, et al. Real-time joint tracking of a hand manipulating an object from RGB-D input[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2016: 294-310.
[19]	BOUKHAYMA A, DE BEM R, TORR P H S. 3D hand shape and pose from images in the wild[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 10835-10844.