Journal of Graphics ›› 2025, Vol. 46 ›› Issue (2): 358-368.DOI: 10.11996/JG.j.2095-302X.2025020358
• Computer Graphics and Virtual Reality • Previous Articles Next Articles
ZHANG Xuhui(
), GUO Yu(
), HUANG Shaohua, ZHENG Guanguan, TANG Pengzhou, MA Xusheng
Received:2024-08-06
Accepted:2024-11-20
Online:2025-04-30
Published:2025-04-24
Contact:
GUO Yu
About author:First author contact:ZHANG Xuhui (1999-), master student. His main research interest covers human-machine collaboration. E-mail:xuhuizhang@nuaa.edu.cn
Supported by:CLC Number:
ZHANG Xuhui, GUO Yu, HUANG Shaohua, ZHENG Guanguan, TANG Pengzhou, MA Xusheng. Grasp pose generation for dexterous hand with integrated knowledge transfer[J]. Journal of Graphics, 2025, 46(2): 358-368.
Add to citation manager EndNote|Ris|BibTeX
URL: http://www.txxb.com.cn/EN/10.11996/JG.j.2095-302X.2025020358
Fig. 10 Mapping rule ((a) Mapping rules based on fingertips; (b) Mapping rules based on fingertips and proximal interphalangeal key points; (c) Optimized mapping rules)
| 名称 | 规格/mm |
|---|---|
| 锤子 | 204×117×23 |
| 锤子2 | 332×130×33 |
| 电钻 | 203×182×57 |
| 电钻2 | 224×174×90 |
Table 1 Tool size
| 名称 | 规格/mm |
|---|---|
| 锤子 | 204×117×23 |
| 锤子2 | 332×130×33 |
| 电钻 | 203×182×57 |
| 电钻2 | 224×174×90 |
| 配置项 | 型号 |
|---|---|
| 编程语言 | Python3.8 |
| 深度学习框架 | Pytorch2.0 |
| 操作系统 | Ubuntu22.04 |
| CPU | Intel(R) Core(TM) i9-10980XE |
| 运行内存 | 128 G |
| GPU | NVIDIA GeForce RTX 3090 |
Table 2 Experimental environment configuration
| 配置项 | 型号 |
|---|---|
| 编程语言 | Python3.8 |
| 深度学习框架 | Pytorch2.0 |
| 操作系统 | Ubuntu22.04 |
| CPU | Intel(R) Core(TM) i9-10980XE |
| 运行内存 | 128 G |
| GPU | NVIDIA GeForce RTX 3090 |
| 指标名称 | 指标评估内容 | 评估方法 |
|---|---|---|
| 手-物互穿体积 | 评估物理合理性 | 通过将网格体素化为1 mm3立方体并计算手表面内部体素体积的总和来作为互穿体积 |
| 仿真位移 | 评估抓取的稳定性 | 将物体和预测的抓取放入模拟器中,并测量物体质心在重力的影响下的平均模拟位移 |
| 手部指间自碰撞 | 评估手部不同区域碰撞情况 | 将手部的三角面片模型划分为6个区域,并将存在连接关系的区域之间的面片进行排除,避免计算碰撞关系时存在歧义,如 |
| 平均最大穿透深度 | 评估灵巧手的抓取质量 | 选取n个抓取姿态,计算映射后的灵巧手与工具的凸包碰撞体之间的平均最大穿透深度 |
| 收敛比例 | 评估映射规则的收敛性 | 选取n个抓取姿态,统计在m次迭代之内,映射函数小于阈值的比例 |
| 抓取姿态的合理性 | 定性评估抓取姿态 | 以训练源数据中不同意图下的抓取姿态为参考,判断生成的抓取姿态是否符合指定的意图,抓取位置是否合适并满足视觉合理性 |
Table 3 Evaluation indicators
| 指标名称 | 指标评估内容 | 评估方法 |
|---|---|---|
| 手-物互穿体积 | 评估物理合理性 | 通过将网格体素化为1 mm3立方体并计算手表面内部体素体积的总和来作为互穿体积 |
| 仿真位移 | 评估抓取的稳定性 | 将物体和预测的抓取放入模拟器中,并测量物体质心在重力的影响下的平均模拟位移 |
| 手部指间自碰撞 | 评估手部不同区域碰撞情况 | 将手部的三角面片模型划分为6个区域,并将存在连接关系的区域之间的面片进行排除,避免计算碰撞关系时存在歧义,如 |
| 平均最大穿透深度 | 评估灵巧手的抓取质量 | 选取n个抓取姿态,计算映射后的灵巧手与工具的凸包碰撞体之间的平均最大穿透深度 |
| 收敛比例 | 评估映射规则的收敛性 | 选取n个抓取姿态,统计在m次迭代之内,映射函数小于阈值的比例 |
| 抓取姿态的合理性 | 定性评估抓取姿态 | 以训练源数据中不同意图下的抓取姿态为参考,判断生成的抓取姿态是否符合指定的意图,抓取位置是否合适并满足视觉合理性 |
| 模型 | 意图 | 穿透 体积/cm3 | 仿真 位移/m | 手部自碰撞 概率/% |
|---|---|---|---|---|
| GraspTTA | Use | 1.235 | 0.012 | 0 |
| Pass | 1.150 | 0.011 | 0 | |
| IntGen | Use | 0.741 | 0.011 | 0 |
| Pass | 0.542 | 0.021 | 0 | |
| IntContact | Use | 0.654 | 0.009 | 12 |
| Pass | 0.398 | 0.016 | 40 |
Table 4 Comparison of grasp pose generation algorithms for power drill
| 模型 | 意图 | 穿透 体积/cm3 | 仿真 位移/m | 手部自碰撞 概率/% |
|---|---|---|---|---|
| GraspTTA | Use | 1.235 | 0.012 | 0 |
| Pass | 1.150 | 0.011 | 0 | |
| IntGen | Use | 0.741 | 0.011 | 0 |
| Pass | 0.542 | 0.021 | 0 | |
| IntContact | Use | 0.654 | 0.009 | 12 |
| Pass | 0.398 | 0.016 | 40 |
| 模型 | 意图 | 穿透 体积/cm3 | 仿真 位移/m | 手部自碰撞 概率/% |
|---|---|---|---|---|
| GraspTTA | Use | 2.692 | 0.017 | 100 |
| Pass | 2.054 | 0.011 | 0 | |
| IntGen | Use | 4.732 | 0.029 | 100 |
| Pass | 1.407 | 0.022 | 0 | |
| IntContact | Use | 1.865 | 0.019 | 0 |
| Pass | 0.719 | 0.012 | 13 |
Table 5 Comparison of grasp pose generation algorithms for power drill
| 模型 | 意图 | 穿透 体积/cm3 | 仿真 位移/m | 手部自碰撞 概率/% |
|---|---|---|---|---|
| GraspTTA | Use | 2.692 | 0.017 | 100 |
| Pass | 2.054 | 0.011 | 0 | |
| IntGen | Use | 4.732 | 0.029 | 100 |
| Pass | 1.407 | 0.022 | 0 | |
| IntContact | Use | 1.865 | 0.019 | 0 |
| Pass | 0.719 | 0.012 | 13 |
Fig. 13 Examples of different algorithms for hammer and drill grasp pose generation under different intents ((a1, a2) GrasspTTA_use; (b1, b2) IntGen_use; (c1, c2) IntContact_use; (d1, d2) GraspTTA_pass; (e1, e2) IntGen_pass; (f1, f2) IntContact_pass)
| 模型 | 意图 | 穿透体积/cm3 | 仿真位移/m | 手部指间自碰撞/% |
|---|---|---|---|---|
| IntContact | Use | 0.900 | 0.015 | 58 |
| Pass | 1.171 | 0.028 | 57 | |
| IntContact+Tink | Use | 0.170 | 0.014 | 35 |
| Pass | 0.280 | 0.023 | 55 | |
| IntContact+Tink+CollisionOurs | Use | 0.169 | 0.013 | 4 |
| Pass | 0.283 | 0.025 | 10 |
Table 6 Hammer_2 grasp posture generation in ablation experiment
| 模型 | 意图 | 穿透体积/cm3 | 仿真位移/m | 手部指间自碰撞/% |
|---|---|---|---|---|
| IntContact | Use | 0.900 | 0.015 | 58 |
| Pass | 1.171 | 0.028 | 57 | |
| IntContact+Tink | Use | 0.170 | 0.014 | 35 |
| Pass | 0.280 | 0.023 | 55 | |
| IntContact+Tink+CollisionOurs | Use | 0.169 | 0.013 | 4 |
| Pass | 0.283 | 0.025 | 10 |
| 模型 | 意图 | 穿透体积/cm3 | 仿真位移/m | 手部指间自碰撞/% |
|---|---|---|---|---|
| IntContact | Use | 2.699 | 0.020 | 37 |
| Pass | 2.836 | 0.021 | 63 | |
| IntContact+Tink | Use | 2.198 | 0.012 | 25 |
| Pass | 1.192 | 0.011 | 12 | |
| IntContact+Tink+CollisionOurs | Use | 2.271 | 0.014 | 0 |
| Pass | 1.212 | 0.011 | 4 |
Table 7 Power drill_2 grasp posture generation in ablation experiment
| 模型 | 意图 | 穿透体积/cm3 | 仿真位移/m | 手部指间自碰撞/% |
|---|---|---|---|---|
| IntContact | Use | 2.699 | 0.020 | 37 |
| Pass | 2.836 | 0.021 | 63 | |
| IntContact+Tink | Use | 2.198 | 0.012 | 25 |
| Pass | 1.192 | 0.011 | 12 | |
| IntContact+Tink+CollisionOurs | Use | 2.271 | 0.014 | 0 |
| Pass | 1.212 | 0.011 | 4 |
Fig. 14 Example of grasp pose of hammer_2 and drill_2 in ablation experiments ((a1, a2) IntContact_use; (b1, b2) IntContact+Tink_use; (c1, c2) Ours_use; (d1, d2) IntContact_pass; (e1, e2) IntContact+Tink_pass; (f1, f2) Ours_pass)
| 映射规则 | 意图 | 平均最大穿透深度/mm | 收敛比例/% |
|---|---|---|---|
| 指尖 | Use | 11.6 | 10 |
| Pass | 17.8 | 52 | |
| 指尖与近侧 指间关键点 | Use | 5.4 | 76 |
| Pass | 10.8 | 94 | |
| 优化 | Use | 5.3 | 90 |
| Pass | 9.9 | 94 |
Table 8 Dexterous hand grasp hammer_2 under different mapping rules
| 映射规则 | 意图 | 平均最大穿透深度/mm | 收敛比例/% |
|---|---|---|---|
| 指尖 | Use | 11.6 | 10 |
| Pass | 17.8 | 52 | |
| 指尖与近侧 指间关键点 | Use | 5.4 | 76 |
| Pass | 10.8 | 94 | |
| 优化 | Use | 5.3 | 90 |
| Pass | 9.9 | 94 |
| 映射规则 | 意图 | 平均最大穿透深度/mm | 收敛比例/% |
|---|---|---|---|
| 指尖 | Use | 29.2 | 0 |
| Pass | 16.6 | 84 | |
| 指尖与近侧 指间关键点 | Use | 4.2 | 100 |
| Pass | 13.9 | 86 | |
| 优化 | Use | 3.1 | 100 |
| Pass | 12.4 | 90 |
Table 9 Dexterous hand grasp drill_2 under different mapping rules
| 映射规则 | 意图 | 平均最大穿透深度/mm | 收敛比例/% |
|---|---|---|---|
| 指尖 | Use | 29.2 | 0 |
| Pass | 16.6 | 84 | |
| 指尖与近侧 指间关键点 | Use | 4.2 | 100 |
| Pass | 13.9 | 86 | |
| 优化 | Use | 3.1 | 100 |
| Pass | 12.4 | 90 |
Fig. 15 Example of dexterous hand grasp hammer_2 and drill_2 under different mapping rules ((a1, a2) A_use; (b1, b2) B_use; (c1, c2) C_use; (d1, d2) A_pass; (e1, e2) B_pass; (f1, f2) C_pass)
Fig. 16 Example of different brands of dexterous hands grasp hammer_2, drill_2 under mapping rule c ((a1, a2) Schunk_use; (b1, b2) Shadow_use; (c1, c2) Ability_use; (d1, d2) Schunk_pass; (e1, e2) Shadow_pass; (f1, f2) Ability_pass)
| [1] |
李泳耀, 江磊, 刘宇飞, 等. 仿人灵巧手的稳定抓取方法研究综述[J]. 兵工学报, 2023, 44(11): 3237-3252.
DOI |
|
LI Y Y, JIANG L, LIU Y F, et al. A review of stable grasping methods for humanoid dexterous hands[J]. Acta Armamentarii, 2023, 44(11): 3237-3252 (in Chinese).
DOI |
|
| [2] | 徐昱琳, 徐粟轩, 徐逍, 等. SHU-II五指仿人灵巧手的运动学及抓取分析[J]. 仪器仪表学报, 2018, 39(9): 30-39. |
| XU Y L, XU S X, XU X, et al. Kinematics and grasping analysis of SHU-II five fingers humanoid dexterous hand[J]. Chinese Journal of Scientific Instrument, 2018, 39(9): 30-39 (in Chinese). | |
| [3] | 童立靖, 李嘉伟. 一种基于改进PointNet++网络的三维手姿估计方法[J]. 图学学报, 2022, 43(5): 892-900. |
| TONG L J, LI J W. A 3D hand pose estimation method based on improved PointNet++[J]. Journal of Graphics, 2022, 43(5): 892-900 (in Chinese). | |
| [4] |
蔡世波, 陶志成, 万伟伟, 等. 机器人多指灵巧手的研究现状、趋势与挑战[J]. 机械工程学报, 2021, 57(15): 1-14.
DOI |
|
CAI S B, TAO Z C, WAN W W, et al. Multi-fingered dexterous hands: from simplicity to complexity and simplifying complex applications[J]. Journal of Mechanical Engineering, 2021, 57(15): 1-14 (in Chinese).
DOI |
|
| [5] | MILLER A T, ALLEN P K. Graspit! a versatile simulator for robotic grasping[J]. IEEE Robotics & Automation Magazine, 2004, 11(4): 110-122. |
| [6] | DZIDEK B M, ADAMS M J, ANDREWS J W, et al. Contact mechanics of the human finger pad under compressive loads[J]. Journal of the Royal Society Interface, 2017, 14(127): 20160935. |
| [7] | 伍一鹤, 张振宁, 仇栋, 等. 基于深度强化学习的虚拟手自适应抓取研究[J]. 图学学报, 2021, 42(3): 462-469. |
| WU Y H, ZHANG Z N, QIU D, et al. Research on adaptive grasping of virtual hands based on deep reinforcement learning[J]. Journal of Graphics, 2021, 42(3): 462-469 (in Chinese). | |
| [8] | ZHU T Q, WU R N, LIN X B, et al. Toward human-like grasp: dexterous grasping via semantic representation of object-hand[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 15721-15731. |
| [9] | LIU S W, ZHOU Y, YANG J M, et al. ContactGen: generative contact modeling for grasp generation[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 20552-20563. |
| [10] | YANG L X, LI K L, ZHAN X Y, et al. OakInk: a large-scale knowledge repository for understanding hand-object interaction[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 20921-20930. |
| [11] | BRAHMBHATT S, TANG C C, TWIGG C D, et al. ContactPose: a dataset of grasps with object contact and hand pose[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 361-378. |
| [12] | TAHERI O, GHORBANI N, BLACK M J, et al. GRAB: a dataset of whole-body human grasping of objects[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 581-600. |
| [13] | SOHN K, YAN X C, LEE H. Learning structured output representation using deep conditional generative models[C]// The 28th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2015: 3483-3491. |
| [14] | QI C R, YI L, SU H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[C]// The 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 5105-5114. |
| [15] | PARK J J, FLORENCE P, STRAUB J, et al. DeepSDF: learning continuous signed distance functions for shape representation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 165-174. |
| [16] | LORENSEN W E, CLINE H E. Marching cubes: a high resolution 3D surface construction algorithm[M]// WOLFER. Seminal Graphics:Pioneering Efforts that Shaped the Field. New York: Association for Computing Machinery, 1998: 347-353. |
| [17] | ROMERO J, TZIONAS D, BLACK M J. Embodied hands: modeling and capturing hands and bodies together[J]. ACM Transactions on Graphics, 2017, 36(6): 245. |
| [18] | XIANG F B, QIN Y Z, MO K C, et al. SAPIEN: a simulated part-based interactive environment[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 11094-11104. |
| [19] | HANDA A, VAN WYK K, YANG W, et al. DexPilot: vision-based teleoperation of dexterous robotic hand-arm system[C]// 2020 IEEE International Conference on Robotics and Automation. New York: IEEE Press, 2020: 9164-9170. |
| [20] | QIN Y Z, WU Y H, LIU S W, et al. DexMV: imitation learning for dexterous manipulation from human videos[C]// The 17th European Conference on Computer Vision. Cham: Springer, 2022: 570-587. |
| [21] | JIANG H W, LIU S W, WANG J S, et al. Hand-object contact consistency reasoning for human grasps generation[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 11087-11096. |
| [1] | CHEN Zhizhang, FENG Yingchaojie, WENG Luoxuan, SHEN Jian, CHEN Wei. DRec: large language model-driven data analysis recommendation system [J]. Journal of Graphics, 2025, 46(5): 1028-1041. |
| [2] | YUE Zijia, WANG Wensong, CHEN Shuangmin, XIN Shiqing, TU Changhe. Geodesic distance propagation across open boundaries [J]. Journal of Graphics, 2025, 46(5): 1042-1049. |
| [3] | ZUO Yuqi, ZHANG Yunfeng, ZHANG Qiuyue, XU Yingcheng. Knowledge-aware recommendation based on hypergraph representation learning and Transformer model optimization [J]. Journal of Graphics, 2025, 46(5): 1050-1060. |
| [4] | WU Haoyu, YANG Xiaochao, WANG Wei, ZHAO Gang. Simulation technology for braiding process of composite materials based on kinematic principles [J]. Journal of Graphics, 2025, 46(5): 1061-1071. |
| [5] | HU Huiqiang, HE Changyan, LIU Xiaojun, JIA Jinyuan, GAO Lu. Towards unsupervised BIM product retrieval: a Weisfeiler-Lehman kernel enhanced approach [J]. Journal of Graphics, 2025, 46(5): 1123-1133. |
| [6] | HUANG Kaiqi, WU Meiqi, CHEN Honghao, FENG Xiaokun, ZHANG Dailing. The three realms of visual turing: from seeing to imagining in the LLM era [J]. Journal of Graphics, 2025, 46(5): 919-930. |
| [7] | HUANG Jing, SHI Ruihao, SONG Wenming, GUO Hepan, WEI Huang, WEI Xiaosong, YAO Jian. A review of autonomous driving image synthesis methods: from simulators to new paradigms [J]. Journal of Graphics, 2025, 46(5): 931-949. |
| [8] | ZHAI Yongjie, ZHAI Bangchao, HU Zhedong, YANG Ke, WANG Qianming, ZHAO Xiaoyu. Adaptive feature fusion pyramid and attention mechanism-based method for transmission line insulator defect detection [J]. Journal of Graphics, 2025, 46(5): 950-959. |
| [9] | LENG Shuo, WANG Wei, OU Jiayong, XUE Zhigang, SONG Yinglong, MO Sijun. On-Site construction safety monitoring based on large vision language models [J]. Journal of Graphics, 2025, 46(5): 960-968. |
| [10] | LIU Cheng, ZHANG Jiayi, YUAN Feng, ZHANG Rui, GAO Xin. SAM2-based multi-objective automatic segmentation method for laparoscopic surgery [J]. Journal of Graphics, 2025, 46(5): 969-979. |
| [11] | YE Wenlong, CHEN Bin. PanoLoRA: an efficient finetuning method for panoramic image generation based on Stable Diffusion [J]. Journal of Graphics, 2025, 46(5): 980-989. |
| [12] | ZHU Hongmiao, ZHONG Guojie, ZHANG Yanci. Semantic segmentation of small-scale point clouds based on integration of mean shift and deep learning [J]. Journal of Graphics, 2025, 46(5): 998-1009. |
| [13] | GUO Ruidong, LAN Guiwen, FAN Donglin, ZHONG Zhan, XU Zirui, REN Xinyue. An object detection algorithm for powerline inspection based on the feature focus & diffusion network [J]. Journal of Graphics, 2025, 46(4): 719-726. |
| [14] | LEI Songlin, ZHAO Zhengpeng, YANG Qiuxia, PU Yuanyuan, GU Jinjing, XU Dan. Zero-shot style transfer based on decoupled diffusion models [J]. Journal of Graphics, 2025, 46(4): 727-738. |
| [15] | CHEN Dong, LI Changlong, DU Zhenlong, SONG Shuang, LI Xiaoli. Intelligent depiction to illumination and shadow: robust video shadow extraction based on SAM [J]. Journal of Graphics, 2025, 46(4): 739-745. |
| Viewed | ||||||
|
Full text |
|
|||||
|
Abstract |
|
|||||