融合生物力学约束与多模态数据的手部重建

doi:10.11996/JG.j.2095-302X.2023040794

图学学报 ›› 2023, Vol. 44 ›› Issue (4): 794-800.DOI: 10.11996/JG.j.2095-302X.2023040794

• 计算机图形学与虚拟现实 • 上一篇下一篇

融合生物力学约束与多模态数据的手部重建

薛皓玮¹(), 王美丽¹^,²^,³()

1.西北农林科技大学信息工程学院，陕西咸阳 712100
2.农业农村部农业物联网重点实验室，陕西咸阳 712100
3.陕西省农业信息感知与智能服务重点实验室，陕西咸阳 712100

收稿日期:2022-11-26 接受日期:2023-04-26 出版日期:2023-08-31 发布日期:2023-08-16
通讯作者: 王美丽(1982-)，女，教授，博士。主要研究方向为计算机图形学、虚拟现实等。E-mail：meili_w@nwsuaf.edu.cn
作者简介:
薛皓玮(2000-)，男，本科生。主要研究方向为计算机视觉和人机交互。E-mail：haowei720@nwafu.edu.cn
基金资助:
陕西省林业科学院2021年科技创新计划专项(SXLK2021-0214);陕西省重点研发项目(2022QFY11-03);农村农业部农业物联网重点实验室项目(2018AIOT-09)

Hand reconstruction incorporating biomechanical constraints and multi-modal data

XUE Hao-wei¹(), WANG Mei-li¹^,²^,³()

1. College of Information Engineering, Northwest A&F University, Xianyang Shaanxi 712100, China
2. Key Laboratory of Agricultural Internet of Things, Ministry of Agriculture and Rural Affairs, Xianyang Shaanxi 712100, China
3. Shaanxi Key Laboratory of Agricultural Information Perception and Intelligence Service, Xianyang Shaanxi 712100, China

Received:2022-11-26 Accepted:2023-04-26 Online:2023-08-31 Published:2023-08-16
Contact: WANG Mei-li (1982-), professor, Ph.D. Her main research interests cover computer graphics, virtual reality, etc. E-mail：meili_w@nwsuaf.edu.cn
About author:
XUE Hao-wei (2000-), undergraduate. His main research interests cover computer vision and human-computer interaction. E-mail：haowei720@nwafu.edu.cn
Supported by:
Shaanxi Academy of Forestry Science 2021 Science and Technology Innovation Program Special(SXLK2021-0214);Shaanxi Province Key Research and Development Program(2022QFY11-03);Key Laboratory Project of Agricultural Internet of Things, Ministry of Rural Agriculture(2018AIOT-09)

摘要/Abstract

摘要：

针对目前单目手部重建设备成本高、响应速度迟缓等问题，提出一种利用单目摄像机获取手部形状和姿态进行手部三维重建的方法。该方法采用基于深度学习的架构，使用带有2D和3D注释的图像数据和手部动作捕捉数据进行训练。首先，通过一个关节检测模块(3DHandNet)和一个逆运动学模块(IRNet)准确地回归3D关节位置并将其映射到关节旋转中。其次，通过引入生物力学约束以达到高质量的网格图像对齐以提供实时预测。最后，将得到的预测向量与关节旋转表示输入到手部网格模板来拟合手部形状。仅与回归3D关节位置的方法相比其更适用于计算机视觉和图形学领域的应用。在基准数据集上的实验结果表明，该方法达到了实时的运行性能(60 fps)和较高的重建精度，在姿态估计精度和手部图像对齐方面均优于目前的方法。

关键词: 手部三维重建, 生物力学约束, 多模态数据, 手部关节检测模块, 逆运动学模块

Abstract:

To address the high cost and slow response of current monocular hand reconstruction, a method for hand 3D reconstruction using a monocular camera to acquire hand shape and posture was proposed. The method adopted a deep learning-based architecture that used image data with 2D and 3D annotations, as well as hand motion capture data for training. Firstly, 3D joint positions were accurately regressed and mapped to joint rotations through a joint detection module (3DHandNet) and an inverse kinematic module (IRNet). Then, biomechanical constraints were introduced to achieve high-quality mesh image alignment for real-time predictions. Finally, the resulting prediction vector with joint rotation representation was input to the hand mesh template to fit the hand shape. This approach was more suitable for computer vision and graphics applications compared to only regressing 3D joint positions. Experimental results on a benchmark dataset demonstrated that the proposed method achieves real-time runtime performance (60 fps) and high reconstruction accuracy, outperforming current methods in terms of hand posture estimation accuracy and hand image alignment.

Key words: 3D reconstruction of the hand, biomechanical constraints, multi-modal data, hand joint detection module, inverse kinematics module

中图分类号:

TP391

薛皓玮, 王美丽. 融合生物力学约束与多模态数据的手部重建[J]. 图学学报, 2023, 44(4): 794-800.

XUE Hao-wei, WANG Mei-li. Hand reconstruction incorporating biomechanical constraints and multi-modal data[J]. Journal of Graphics, 2023, 44(4): 794-800.

图/表 10

图1 手部三维重建结果展示((a)输入图像1；(b)输出结果1；(c) Mesh覆盖1；(d)输入图像2；(e)输出结果2；(f) Mesh覆盖2)

Fig. 1 Display of 3D hand reconstruction results ((a) Input image 1; (b) Output result 1; (c) Mesh overlay 1; (d) Input image 2; (e) Output result 2; (f) Mesh overlay 2)

图2 模型架构概述

Fig. 2 Overview of our architecture

图3 构建的3DHandData数据集的可视化((a)各个角度；(b)真实3D关节位置)

Fig. 3 Visualization of the constructed 3DHandData dataset ((a) All angles; (b) Real 3D joint position)

图4 生物力学结构说明

Fig. 4 Description of biomechajiegounical structure

图5 几个具有挑战性的场景下的结果((a)输入图像；(b)输出结果；(c) Mesh覆盖)

Fig. 5 Show our results for several challenging scenarios ((a) Input image; (b) Output result; (c) Mesh overlay)

图6 三维重建的形状与对齐效果((a)输入图像；(b) 2D位姿；(c) 3D位姿；(d)输出结果；(e) Mesh覆盖)

Fig. 6 Shape and alignment effects of 3D reconstruction ((a) Input image; (b) 2D pose; (c) 3D pose; (d) Output result; (e) Mesh overlay)

图7 与文献[3]方法和文献[15]方法的比较((a)输入图像；(b)文献[15]；(c)文献[3]；(d)本文方法)

Fig. 7 Comparison with the methods in literature [3] and [15] ((a) Input image; (b) Literature [15]; (c) Literature [3]; (d) Ours)

表1 在4个公共数据集上与其他方法进行了比较

Table 1 Comparisons were made with state-of-the-art methods on four public datasets

Method	AUC of PCK
Method	DO	ED	STB	RHD
本文方法	0.953	0.823	0.896	0.950(*)
文献[7]	0.948	0.811	0.898(*)	0.856(*)
文献[3]	-	-	0.998(*)	0.920(*)
文献[4]	0.825	-	0.995(*)	0.901(*)
文献[17]	-	-	0.996(*)	0.943(*)
文献[11]	0.650	-	0.995(*)	0.926(*)
文献[8]	0.912	-	0.994(*)	-
文献[16]	0.763	0.674	0.994(*)	-
文献[5]	0.672	0.543	0.994(*)	-
文献[19]	0.573	-	0.948(*)	0.670(*)

表2 消融实验

Table 2 Ablation study

序号	Variants of our method	AUC of PCK
序号	Variants of our method	DO	ED	RHD
1	Full	0.953±0	0.823±0	0.950±0
2	w/o IRNet	0.913±0	0.802±0.04	0.808±0.07
3	w/o BMC	0.930±5e⁻⁴	0.752±5e⁻⁴	0.925±0.04
4	w/o MSSGesture	0.926±0.05	0.823±0.10	0.809±0.11
5	w/o 3DHandData	0.873±0	0.712±0.02	0.738±0.04

图8 作为基线，与文献[4]进行了比较

Fig. 8 As a baseline, was compared with the literature [4]

参考文献 19

[1]	童立靖, 李嘉伟. 一种基于改进PointNet++网络的三维手姿估计方法[J]. 图学学报, 2022, 43(5): 892-900.
	TONG L J, LI J W. A 3D hand pose estimation method based on improved PointNet++[J]. Journal of Graphics, 2022, 43(5): 892-900 (in Chinese).
[2]	TANG X, WANG T Y, FU C W. Towards accurate alignment in real-time 3D hand-mesh reconstruction[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 11698-11707.
[3]	GE L H, REN Z, LI Y C, et al. 3D hand shape and pose estimation from a single RGB image[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 10825-10834.
[4]	ZHANG X, LI Q, MO H, et al. End-to-end hand mesh recovery from a monocular RGB image[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2020: 2354-2364.
[5]	IQBAL U, MOLCHANOV P, BREUEL T, et al. Hand pose estimation via latent 2.5D heatmap regression[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 125-143.
[6]	CAI Y J, GE L H, CAI J F, et al. Weakly-supervised 3D hand pose estimation from monocular RGB images[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 678-694.
[7]	ROMERO J, TZIONAS D, BLACK M J. Embodied hands: modeling and capturing hands and bodies together[J]. ACM Transactions on Graphics, 2017, 36(6): 245. 1-245.17.
[8]	ZHOU Y X, HABERMANN M, XU W P, et al. Monocular real-time hand shape and motion capture using multi-modal data[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 5345-5354.
[9]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 770-778.
[10]	ARMAGAN A, GARCIA-HERNANDO G, BAEK S, et al. Measuringgeneralisation to unseen viewpoints, articulations, shapes and objects for 3D hand pose estimation under hand-object interaction[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 85-101.
[11]	BAEK S, KIM K I, KIM T K. Pushing the envelope for RGB-based dense 3D hand pose estimation via neural rendering[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1067-1076.
[12]	ALBRECHT I, HABER J, SEIDEL H P. Construction and animation of anatomically based human hand models[C]// 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Cham: Springer International Publishing, 2003: 98-109.
[13]	SPURR A, IQBAL U, MOLCHANOV P, et al. Weakly supervised 3D hand pose estimation via biomechanical constraints[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 211-228.
[14]	范晶晶, 薛皓玮, 吴欣鸿, 等. 引入重影特征映射和通道注意力机制的手势识别算法[J]. 计算机辅助设计与图形学学报, 2022, 34(3): 403-414.
	FAN J J, XUE H W, WU X H, et al. Gesture recognition algorithm introducing ghost feature mapping and channel attention mechanism[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(3): 403-414 (in Chinese).
[15]	HASSON Y, VAROL G, TZIONAS D, et al. Learning joint reconstruction of hands and manipulated objects[EB/OL]. [2022-06-15]. https://arxiv.org/abs/1904.05767.
[16]	ZIMMERMANN C, BROX T. Learning to estimate 3D hand pose from single RGB images[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 4913-4921.
[17]	ZHANG J, JIAO J, CHEN M, et al. 3D hand pose tracking and estimation using stereo matching[EB/OL]. [2022-06-15]. https://arxiv.org/abs/1610.07214v1.
[18]	SRIDHAR S, MUELLER F, ZOLLHÖFER M, et al. Real-time joint tracking of a hand manipulating an object from RGB-D input[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2016: 294-310.
[19]	BOUKHAYMA A, DE BEM R, TORR P H S. 3D hand shape and pose from images in the wild[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 10835-10844.

融合生物力学约束与多模态数据的手部重建

Hand reconstruction incorporating biomechanical constraints and multi-modal data

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 19

相关文章 15

编辑推荐

Metrics

本文评价

[1]	闫善武, 肖洪兵, 王瑜, 孙梅. 融合行人时空信息的视频异常检测[J]. 图学学报, 2023, 44(1): 95-103.
[2]	谷雨, 赵军. 列车闸瓦钎及闸瓦故障图像检测算法研究[J]. 图学学报, 2023, 44(1): 88-94.
[3]	张晨阳, 曹艳华, 杨晓忠. 基于分数阶小波与引导滤波的多聚焦图像融合方法[J]. 图学学报, 2023, 44(1): 77-87.
[4]	邵英杰, 尹辉, 谢颖, 黄华. 草图引导的选择循环推理式人脸图像修复网络[J]. 图学学报, 2023, 44(1): 67-76.
[5]	潘森垒, 钱文华, 曹进德, 徐丹. 基于注意力机制的东巴画情感分类[J]. 图学学报, 2023, 44(1): 59-66.
[6]	单芳湄, 王梦文, 李敏. 融合注意力机制的肠道息肉分割多尺度卷积神经网络[J]. 图学学报, 2023, 44(1): 50-58.
[7]	张倩, 王夏黎, 王炜昊, 武历展, 李超. 基于多尺度特征融合的细胞计数方法[J]. 图学学报, 2023, 44(1): 41-49.
[8]	邵文斌, 刘玉杰, 孙晓瑞, 李宗民. 基于残差增强注意力的跨模态行人重识别[J]. 图学学报, 2023, 44(1): 33-40.
[9]	皮骏, 刘宇恒, 李久昊. 基于YOLOv5s的轻量化森林火灾检测算法研究[J]. 图学学报, 2023, 44(1): 26-32.
[10]	裴卉宁, 邵星辰, 谭昭芸, 黄雪芹, 白仲航. 融合DE-GWO与SVR的文化意象预测模型[J]. 图学学报, 2023, 44(1): 184-193.
[11]	范震, 刘晓静, 李小波, 崔亚超. 一种对光照和遮挡鲁棒的单应性估计方法[J]. 图学学报, 2023, 44(1): 166-176.
[12]	李小波, 李阳贵, 郭宁, 范震. 融合注意力机制的YOLOv5口罩检测算法[J]. 图学学报, 2023, 44(1): 16-25.
[13]	刘振晔, 陈仁杰, 刘利刚. 基于边长的三维形状插值[J]. 图学学报, 2023, 44(1): 158-165.
[14]	王佳栋, 曹娟, 陈中贵. 保特征的点云骨架提取算法[J]. 图学学报, 2023, 44(1): 146-157.
[15]	王玉萍, 曾毅, 李胜辉, 张磊. 一种基于Transformer的三维人体姿态估计方法[J]. 图学学报, 2023, 44(1): 139-145.