Journal of Graphics ›› 2025, Vol. 46 ›› Issue (2): 358-368.DOI: 10.11996/JG.j.2095-302X.2025020358
• Computer Graphics and Virtual Reality • Previous Articles Next Articles
ZHANG Xuhui(), GUO Yu(
), HUANG Shaohua, ZHENG Guanguan, TANG Pengzhou, MA Xusheng
Received:
2024-08-06
Accepted:
2024-11-20
Online:
2025-04-30
Published:
2025-04-24
Contact:
GUO Yu
About author:
First author contact:ZHANG Xuhui (1999-), master student. His main research interest covers human-machine collaboration. E-mail:xuhuizhang@nuaa.edu.cn
Supported by:
CLC Number:
ZHANG Xuhui, GUO Yu, HUANG Shaohua, ZHENG Guanguan, TANG Pengzhou, MA Xusheng. Grasp pose generation for dexterous hand with integrated knowledge transfer[J]. Journal of Graphics, 2025, 46(2): 358-368.
Add to citation manager EndNote|Ris|BibTeX
URL: http://www.txxb.com.cn/EN/10.11996/JG.j.2095-302X.2025020358
Fig. 10 Mapping rule ((a) Mapping rules based on fingertips; (b) Mapping rules based on fingertips and proximal interphalangeal key points; (c) Optimized mapping rules)
名称 | 规格/mm |
---|---|
锤子 | 204×117×23 |
锤子2 | 332×130×33 |
电钻 | 203×182×57 |
电钻2 | 224×174×90 |
Table 1 Tool size
名称 | 规格/mm |
---|---|
锤子 | 204×117×23 |
锤子2 | 332×130×33 |
电钻 | 203×182×57 |
电钻2 | 224×174×90 |
配置项 | 型号 |
---|---|
编程语言 | Python3.8 |
深度学习框架 | Pytorch2.0 |
操作系统 | Ubuntu22.04 |
CPU | Intel(R) Core(TM) i9-10980XE |
运行内存 | 128 G |
GPU | NVIDIA GeForce RTX 3090 |
Table 2 Experimental environment configuration
配置项 | 型号 |
---|---|
编程语言 | Python3.8 |
深度学习框架 | Pytorch2.0 |
操作系统 | Ubuntu22.04 |
CPU | Intel(R) Core(TM) i9-10980XE |
运行内存 | 128 G |
GPU | NVIDIA GeForce RTX 3090 |
指标名称 | 指标评估内容 | 评估方法 |
---|---|---|
手-物互穿体积 | 评估物理合理性 | 通过将网格体素化为1 mm3立方体并计算手表面内部体素体积的总和来作为互穿体积 |
仿真位移 | 评估抓取的稳定性 | 将物体和预测的抓取放入模拟器中,并测量物体质心在重力的影响下的平均模拟位移 |
手部指间自碰撞 | 评估手部不同区域碰撞情况 | 将手部的三角面片模型划分为6个区域,并将存在连接关系的区域之间的面片进行排除,避免计算碰撞关系时存在歧义,如 |
平均最大穿透深度 | 评估灵巧手的抓取质量 | 选取n个抓取姿态,计算映射后的灵巧手与工具的凸包碰撞体之间的平均最大穿透深度 |
收敛比例 | 评估映射规则的收敛性 | 选取n个抓取姿态,统计在m次迭代之内,映射函数小于阈值的比例 |
抓取姿态的合理性 | 定性评估抓取姿态 | 以训练源数据中不同意图下的抓取姿态为参考,判断生成的抓取姿态是否符合指定的意图,抓取位置是否合适并满足视觉合理性 |
Table 3 Evaluation indicators
指标名称 | 指标评估内容 | 评估方法 |
---|---|---|
手-物互穿体积 | 评估物理合理性 | 通过将网格体素化为1 mm3立方体并计算手表面内部体素体积的总和来作为互穿体积 |
仿真位移 | 评估抓取的稳定性 | 将物体和预测的抓取放入模拟器中,并测量物体质心在重力的影响下的平均模拟位移 |
手部指间自碰撞 | 评估手部不同区域碰撞情况 | 将手部的三角面片模型划分为6个区域,并将存在连接关系的区域之间的面片进行排除,避免计算碰撞关系时存在歧义,如 |
平均最大穿透深度 | 评估灵巧手的抓取质量 | 选取n个抓取姿态,计算映射后的灵巧手与工具的凸包碰撞体之间的平均最大穿透深度 |
收敛比例 | 评估映射规则的收敛性 | 选取n个抓取姿态,统计在m次迭代之内,映射函数小于阈值的比例 |
抓取姿态的合理性 | 定性评估抓取姿态 | 以训练源数据中不同意图下的抓取姿态为参考,判断生成的抓取姿态是否符合指定的意图,抓取位置是否合适并满足视觉合理性 |
模型 | 意图 | 穿透 体积/cm3 | 仿真 位移/m | 手部自碰撞 概率/% |
---|---|---|---|---|
GraspTTA | Use | 1.235 | 0.012 | 0 |
Pass | 1.150 | 0.011 | 0 | |
IntGen | Use | 0.741 | 0.011 | 0 |
Pass | 0.542 | 0.021 | 0 | |
IntContact | Use | 0.654 | 0.009 | 12 |
Pass | 0.398 | 0.016 | 40 |
Table 4 Comparison of grasp pose generation algorithms for power drill
模型 | 意图 | 穿透 体积/cm3 | 仿真 位移/m | 手部自碰撞 概率/% |
---|---|---|---|---|
GraspTTA | Use | 1.235 | 0.012 | 0 |
Pass | 1.150 | 0.011 | 0 | |
IntGen | Use | 0.741 | 0.011 | 0 |
Pass | 0.542 | 0.021 | 0 | |
IntContact | Use | 0.654 | 0.009 | 12 |
Pass | 0.398 | 0.016 | 40 |
模型 | 意图 | 穿透 体积/cm3 | 仿真 位移/m | 手部自碰撞 概率/% |
---|---|---|---|---|
GraspTTA | Use | 2.692 | 0.017 | 100 |
Pass | 2.054 | 0.011 | 0 | |
IntGen | Use | 4.732 | 0.029 | 100 |
Pass | 1.407 | 0.022 | 0 | |
IntContact | Use | 1.865 | 0.019 | 0 |
Pass | 0.719 | 0.012 | 13 |
Table 5 Comparison of grasp pose generation algorithms for power drill
模型 | 意图 | 穿透 体积/cm3 | 仿真 位移/m | 手部自碰撞 概率/% |
---|---|---|---|---|
GraspTTA | Use | 2.692 | 0.017 | 100 |
Pass | 2.054 | 0.011 | 0 | |
IntGen | Use | 4.732 | 0.029 | 100 |
Pass | 1.407 | 0.022 | 0 | |
IntContact | Use | 1.865 | 0.019 | 0 |
Pass | 0.719 | 0.012 | 13 |
Fig. 13 Examples of different algorithms for hammer and drill grasp pose generation under different intents ((a1, a2) GrasspTTA_use; (b1, b2) IntGen_use; (c1, c2) IntContact_use; (d1, d2) GraspTTA_pass; (e1, e2) IntGen_pass; (f1, f2) IntContact_pass)
模型 | 意图 | 穿透体积/cm3 | 仿真位移/m | 手部指间自碰撞/% |
---|---|---|---|---|
IntContact | Use | 0.900 | 0.015 | 58 |
Pass | 1.171 | 0.028 | 57 | |
IntContact+Tink | Use | 0.170 | 0.014 | 35 |
Pass | 0.280 | 0.023 | 55 | |
IntContact+Tink+CollisionOurs | Use | 0.169 | 0.013 | 4 |
Pass | 0.283 | 0.025 | 10 |
Table 6 Hammer_2 grasp posture generation in ablation experiment
模型 | 意图 | 穿透体积/cm3 | 仿真位移/m | 手部指间自碰撞/% |
---|---|---|---|---|
IntContact | Use | 0.900 | 0.015 | 58 |
Pass | 1.171 | 0.028 | 57 | |
IntContact+Tink | Use | 0.170 | 0.014 | 35 |
Pass | 0.280 | 0.023 | 55 | |
IntContact+Tink+CollisionOurs | Use | 0.169 | 0.013 | 4 |
Pass | 0.283 | 0.025 | 10 |
模型 | 意图 | 穿透体积/cm3 | 仿真位移/m | 手部指间自碰撞/% |
---|---|---|---|---|
IntContact | Use | 2.699 | 0.020 | 37 |
Pass | 2.836 | 0.021 | 63 | |
IntContact+Tink | Use | 2.198 | 0.012 | 25 |
Pass | 1.192 | 0.011 | 12 | |
IntContact+Tink+CollisionOurs | Use | 2.271 | 0.014 | 0 |
Pass | 1.212 | 0.011 | 4 |
Table 7 Power drill_2 grasp posture generation in ablation experiment
模型 | 意图 | 穿透体积/cm3 | 仿真位移/m | 手部指间自碰撞/% |
---|---|---|---|---|
IntContact | Use | 2.699 | 0.020 | 37 |
Pass | 2.836 | 0.021 | 63 | |
IntContact+Tink | Use | 2.198 | 0.012 | 25 |
Pass | 1.192 | 0.011 | 12 | |
IntContact+Tink+CollisionOurs | Use | 2.271 | 0.014 | 0 |
Pass | 1.212 | 0.011 | 4 |
Fig. 14 Example of grasp pose of hammer_2 and drill_2 in ablation experiments ((a1, a2) IntContact_use; (b1, b2) IntContact+Tink_use; (c1, c2) Ours_use; (d1, d2) IntContact_pass; (e1, e2) IntContact+Tink_pass; (f1, f2) Ours_pass)
映射规则 | 意图 | 平均最大穿透深度/mm | 收敛比例/% |
---|---|---|---|
指尖 | Use | 11.6 | 10 |
Pass | 17.8 | 52 | |
指尖与近侧 指间关键点 | Use | 5.4 | 76 |
Pass | 10.8 | 94 | |
优化 | Use | 5.3 | 90 |
Pass | 9.9 | 94 |
Table 8 Dexterous hand grasp hammer_2 under different mapping rules
映射规则 | 意图 | 平均最大穿透深度/mm | 收敛比例/% |
---|---|---|---|
指尖 | Use | 11.6 | 10 |
Pass | 17.8 | 52 | |
指尖与近侧 指间关键点 | Use | 5.4 | 76 |
Pass | 10.8 | 94 | |
优化 | Use | 5.3 | 90 |
Pass | 9.9 | 94 |
映射规则 | 意图 | 平均最大穿透深度/mm | 收敛比例/% |
---|---|---|---|
指尖 | Use | 29.2 | 0 |
Pass | 16.6 | 84 | |
指尖与近侧 指间关键点 | Use | 4.2 | 100 |
Pass | 13.9 | 86 | |
优化 | Use | 3.1 | 100 |
Pass | 12.4 | 90 |
Table 9 Dexterous hand grasp drill_2 under different mapping rules
映射规则 | 意图 | 平均最大穿透深度/mm | 收敛比例/% |
---|---|---|---|
指尖 | Use | 29.2 | 0 |
Pass | 16.6 | 84 | |
指尖与近侧 指间关键点 | Use | 4.2 | 100 |
Pass | 13.9 | 86 | |
优化 | Use | 3.1 | 100 |
Pass | 12.4 | 90 |
Fig. 15 Example of dexterous hand grasp hammer_2 and drill_2 under different mapping rules ((a1, a2) A_use; (b1, b2) B_use; (c1, c2) C_use; (d1, d2) A_pass; (e1, e2) B_pass; (f1, f2) C_pass)
Fig. 16 Example of different brands of dexterous hands grasp hammer_2, drill_2 under mapping rule c ((a1, a2) Schunk_use; (b1, b2) Shadow_use; (c1, c2) Ability_use; (d1, d2) Schunk_pass; (e1, e2) Shadow_pass; (f1, f2) Ability_pass)
[1] |
李泳耀, 江磊, 刘宇飞, 等. 仿人灵巧手的稳定抓取方法研究综述[J]. 兵工学报, 2023, 44(11): 3237-3252.
DOI |
LI Y Y, JIANG L, LIU Y F, et al. A review of stable grasping methods for humanoid dexterous hands[J]. Acta Armamentarii, 2023, 44(11): 3237-3252 (in Chinese).
DOI |
|
[2] | 徐昱琳, 徐粟轩, 徐逍, 等. SHU-II五指仿人灵巧手的运动学及抓取分析[J]. 仪器仪表学报, 2018, 39(9): 30-39. |
XU Y L, XU S X, XU X, et al. Kinematics and grasping analysis of SHU-II five fingers humanoid dexterous hand[J]. Chinese Journal of Scientific Instrument, 2018, 39(9): 30-39 (in Chinese). | |
[3] | 童立靖, 李嘉伟. 一种基于改进PointNet++网络的三维手姿估计方法[J]. 图学学报, 2022, 43(5): 892-900. |
TONG L J, LI J W. A 3D hand pose estimation method based on improved PointNet++[J]. Journal of Graphics, 2022, 43(5): 892-900 (in Chinese). | |
[4] |
蔡世波, 陶志成, 万伟伟, 等. 机器人多指灵巧手的研究现状、趋势与挑战[J]. 机械工程学报, 2021, 57(15): 1-14.
DOI |
CAI S B, TAO Z C, WAN W W, et al. Multi-fingered dexterous hands: from simplicity to complexity and simplifying complex applications[J]. Journal of Mechanical Engineering, 2021, 57(15): 1-14 (in Chinese).
DOI |
|
[5] | MILLER A T, ALLEN P K. Graspit! a versatile simulator for robotic grasping[J]. IEEE Robotics & Automation Magazine, 2004, 11(4): 110-122. |
[6] | DZIDEK B M, ADAMS M J, ANDREWS J W, et al. Contact mechanics of the human finger pad under compressive loads[J]. Journal of the Royal Society Interface, 2017, 14(127): 20160935. |
[7] | 伍一鹤, 张振宁, 仇栋, 等. 基于深度强化学习的虚拟手自适应抓取研究[J]. 图学学报, 2021, 42(3): 462-469. |
WU Y H, ZHANG Z N, QIU D, et al. Research on adaptive grasping of virtual hands based on deep reinforcement learning[J]. Journal of Graphics, 2021, 42(3): 462-469 (in Chinese). | |
[8] | ZHU T Q, WU R N, LIN X B, et al. Toward human-like grasp: dexterous grasping via semantic representation of object-hand[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 15721-15731. |
[9] | LIU S W, ZHOU Y, YANG J M, et al. ContactGen: generative contact modeling for grasp generation[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 20552-20563. |
[10] | YANG L X, LI K L, ZHAN X Y, et al. OakInk: a large-scale knowledge repository for understanding hand-object interaction[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 20921-20930. |
[11] | BRAHMBHATT S, TANG C C, TWIGG C D, et al. ContactPose: a dataset of grasps with object contact and hand pose[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 361-378. |
[12] | TAHERI O, GHORBANI N, BLACK M J, et al. GRAB: a dataset of whole-body human grasping of objects[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 581-600. |
[13] | SOHN K, YAN X C, LEE H. Learning structured output representation using deep conditional generative models[C]// The 28th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2015: 3483-3491. |
[14] | QI C R, YI L, SU H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[C]// The 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 5105-5114. |
[15] | PARK J J, FLORENCE P, STRAUB J, et al. DeepSDF: learning continuous signed distance functions for shape representation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 165-174. |
[16] | LORENSEN W E, CLINE H E. Marching cubes: a high resolution 3D surface construction algorithm[M]// WOLFER. Seminal Graphics:Pioneering Efforts that Shaped the Field. New York: Association for Computing Machinery, 1998: 347-353. |
[17] | ROMERO J, TZIONAS D, BLACK M J. Embodied hands: modeling and capturing hands and bodies together[J]. ACM Transactions on Graphics, 2017, 36(6): 245. |
[18] | XIANG F B, QIN Y Z, MO K C, et al. SAPIEN: a simulated part-based interactive environment[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 11094-11104. |
[19] | HANDA A, VAN WYK K, YANG W, et al. DexPilot: vision-based teleoperation of dexterous robotic hand-arm system[C]// 2020 IEEE International Conference on Robotics and Automation. New York: IEEE Press, 2020: 9164-9170. |
[20] | QIN Y Z, WU Y H, LIU S W, et al. DexMV: imitation learning for dexterous manipulation from human videos[C]// The 17th European Conference on Computer Vision. Cham: Springer, 2022: 570-587. |
[21] | JIANG H W, LIU S W, WANG J S, et al. Hand-object contact consistency reasoning for human grasps generation[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 11087-11096. |
[1] | WANG Zhidong, CHEN Chenyang, LIU Xiaoming. Defect detection method of communication optical cable based on adaptive feature extraction [J]. Journal of Graphics, 2025, 46(2): 241-248. |
[2] | LI Zhihuan, NING Xiaojuan, LV Zhiyong, SHI Zhenghao, JIN Haiyan, WANG Yinghui, ZHOU Wenming. DEMF-Net: dual-branch feature enhancement and multi-scale fusion for semantic segmentation of large-scale point clouds [J]. Journal of Graphics, 2025, 46(2): 259-269. |
[3] | WANG Xueting, GUO Xin, WANG Song, CHEN Enqing. Human skeleton action recognition method based on variational autoencoder masked reconstruction [J]. Journal of Graphics, 2025, 46(2): 270-278. |
[4] | GUO Yecai, HU Xiaowei, AMITAVE Saha, MAO Xiangnan. Multiscale dense interactive attention residual real image denoising network [J]. Journal of Graphics, 2025, 46(2): 279-287. |
[5] | ZHAI Yongjie, WANG Luyao, ZHAO Xiaoyu, HU Zhedong, WANG Qianming, WANG Yaru. Multi-fitting detection for transmission lines based on a cascade query-position relationship method [J]. Journal of Graphics, 2025, 46(2): 288-299. |
[6] | LIU Gaoyi, HU Ruizhen, LIU Ligang. 3D Gaussian splatting semantic segmentation and editing based on 2D feature distillation [J]. Journal of Graphics, 2025, 46(2): 312-321. |
[7] | ZHANG Tiansheng, ZHU Minfeng, REN Yiwen, WANG Chenhan, ZHANG Lidong, ZHANG Wei, CHEN Wei. BPA-SAM: box prompt augmented SAM for traditional Chinese realistic painting [J]. Journal of Graphics, 2025, 46(2): 322-331. |
[8] | SUN Heyi, LI Yixiao, TIAN Xi, ZHANG Songhai. Image to 3D vase generation technology combining procedural content generation and diffusion models [J]. Journal of Graphics, 2025, 46(2): 332-344. |
[9] | ZHU Xiaoqiang, YANG Yifei. Free sculpting system in virtual reality environment [J]. Journal of Graphics, 2025, 46(2): 345-357. |
[10] | ZHOU Wei, CANG Minnan, CHENG Haozong. Research on the method of 3D image reconstruction for cultural relics based on AR technology [J]. Journal of Graphics, 2025, 46(2): 369-381. |
[11] | LI Jiyuan, GUAN Zheyu, SONG Haichuan, TAN Xin, MA Lizhuang. Human-in-the-loop field-specific logo generation method [J]. Journal of Graphics, 2025, 46(2): 382-392. |
[12] | FANG Chenghao, WANG Kangkan. 3D human pose and shape estimation from single-view point clouds with semi-supervised learning [J]. Journal of Graphics, 2025, 46(2): 393-401. |
[13] | QIU Jiaxin, SONG Qianyun, XU Dan. A neural radiation field-based approach to ethnic dance reconstruction [J]. Journal of Graphics, 2025, 46(2): 415-424. |
[14] | LIU Shengjun, TAO Shanshan, WANG Haibo, LI Qinsong, LIU Xinru. High-precision reconstruction of swept surfaces with a planar path [J]. Journal of Graphics, 2025, 46(2): 425-436. |
[15] | WANG Wensong, ZHOU Zijun, XIN Shiqing, TU Changhe, WANG Wenping. Tracing high-quality isolines for discrete geodesic distance fields [J]. Journal of Graphics, 2025, 46(2): 437-448. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||