融合知识迁移的灵巧手抓取姿态生成

doi:10.11996/JG.j.2095-302X.2025020358

图学学报 ›› 2025, Vol. 46 ›› Issue (2): 358-368.DOI: 10.11996/JG.j.2095-302X.2025020358

• 计算机图形学与虚拟现实 • 上一篇下一篇

融合知识迁移的灵巧手抓取姿态生成

张旭辉(), 郭宇(), 黄少华, 郑冠冠, 汤鹏洲, 马旭升

南京航空航天大学机电学院，江苏南京 210016

收稿日期:2024-08-06 接受日期:2024-11-20 出版日期:2025-04-30 发布日期:2025-04-24
通讯作者:郭宇(1971-)，男，教授，博士。主要研究方向为增强装配与人机协作等。E-mail：guoyu@nuaa.edu.cn
第一作者:张旭辉(1999-)，男，硕士研究生。主要研究方向为人机协作。E-mail：xuhuizhang@nuaa.edu.cn
基金资助:
南京市重大科技专项(202309018)

Grasp pose generation for dexterous hand with integrated knowledge transfer

ZHANG Xuhui(), GUO Yu(), HUANG Shaohua, ZHENG Guanguan, TANG Pengzhou, MA Xusheng

College of Mechanical & Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing Jiangsu 210016, China

Received:2024-08-06 Accepted:2024-11-20 Published:2025-04-30 Online:2025-04-24
Contact: GUO Yu (1971-), professor, Ph.D. His main research interests cover augmented assembly, human-machine collaboration etc. E-mail：guoyu@nuaa.edu.cn
First author：ZHANG Xuhui (1999-), master student. His main research interest covers human-machine collaboration. E-mail：xuhuizhang@nuaa.edu.cn
Supported by:
Nanjing Major Science and Technology Projects(202309018)

摘要/Abstract

摘要：

五指灵巧手抓取姿态的生成在灵巧手抓取任务上具有重要意义。首先，针对不同使用意图下人手对工具的抓取姿态不同的特点，构建了基于意图的抓取姿态生成网络，强调了不同意图下抓取的功能性；其次，针对在有限的数据下训练的抓取姿态生成网络无法适应所有类内工具的问题，提出了一种融合知识迁移的抓取姿态生成方法，改进知识迁移方法以适应各种姿态的类内目标工具以生成功能性抓取，同时优化手部指间自碰撞问题；最终，在构建人手与五指灵巧手的抓取姿态映射关系时，优化基于关键点对应关系的映射规则，实现了五指灵巧手在不同意图下对类内目标工具的抓取姿态生成，为工具的后续使用操作打好基础。通过基于意图的抓取姿态生成与知识迁移相结合的方法，使得在有限数据训练得到的基于意图的抓取姿态生成网络，可以对类内目标工具生成更好的抓取姿态，相较于原网络针对实验中的类内目标工具在穿透体积上平均降低0.917 cm³，仿真位移平均降低5.25 mm，手部指间自碰撞概率平均降低49.25%。

关键词: 五指灵巧手, 抓取姿态生成, 知识迁移, 手部指间自碰撞, 抓取姿态映射

Abstract:

Grasp pose generation for a five-finger dexterous hand plays a critical role in dexterous hand grasping tasks. Firstly, an intention-based grasp pose generation network was constructed, addressing variations in human hand grasping poses under different tool usage intentions, emphasizing the functionality of grasps under different intentions. Secondly, to tackle the issue that the a grasp pose generation network trained with limited data cannot adapt to all intra-class tools, a knowledge transfer-based grasp pose generation method was proposed. This method improved knowledge transfer to adapt to various poses of intra-class target tools for functional grasp while optimizing the inter-finger self-collision issue. Finally, in constructing the mapping relationship between the human hand and five-finger dexterous hand grasp poses, key point correspondence-based mapping rules were optimized. This enabled the generation of five-finger dexterous hand grasp poses under different intentions, laying a foundation for subsequent tool use operations. By combining intention-based grasp pose generation with knowledge transfer, the intention-based grasp pose generation network trained with limited data can generate better grasp poses for intra-class target tools. Compared to the original network, the proposed method reduced the penetration volume by an average of 0.917 cm3, the simulation displacement by an average of 5.25 mm, and the inter-finger self-collision probability by an average of 49.25%.

Key words: five-finger dexterous hand, grasp pose generation, knowledge transfer, inter-finger self-collision, grasp mapping

中图分类号:

TP391
TP242

张旭辉, 郭宇, 黄少华, 郑冠冠, 汤鹏洲, 马旭升. 融合知识迁移的灵巧手抓取姿态生成[J]. 图学学报, 2025, 46(2): 358-368.

ZHANG Xuhui, GUO Yu, HUANG Shaohua, ZHENG Guanguan, TANG Pengzhou, MA Xusheng. Grasp pose generation for dexterous hand with integrated knowledge transfer[J]. Journal of Graphics, 2025, 46(2): 358-368.

图/表 27

图1 末端执行器((a)平行夹爪；(b)灵巧手)

Fig. 1 End effector ((a) Parallel jaws; (b) Dexterous hand)

图2 不同意图下对锤子的抓取姿态((a)使用意图；(b)递出意图)

Fig. 2 Grasp pose for hammer with different intentions ((a) Use-intention; (b) Pass-intention)

图3 融合知识迁移的灵巧手抓取姿态生成算法

Fig. 3 The algorithms of grasp pose generation for dexterous hands with integrated knowledge transfer

图4 IntContact网络结构

Fig. 4 The network structure of IntContact

图5 类内工具配准((a)初始点云；(b)配准后点云)

Fig. 5 Intra-class instrument alignment ((a) Primordial point cloud; (b) Post-aligned point cloud)

图6 隐式形状插值

Fig. 6 Implicit shape interpolation

图7 显性接触图映射

Fig. 7 Explicit contact mapping

图8 迭代姿态细化求解抓取姿态

Fig. 8 Iterative pose refinement solves for grasp pose

图9 MANO手部模型

Fig. 9 MANO hand model

图10 映射规则((a)基于指尖的映射规则；(b)基于指尖与近侧指间关键点的映射规则；(c)优化映射规则)

Fig. 10 Mapping rule ((a) Mapping rules based on fingertips; (b) Mapping rules based on fingertips and proximal interphalangeal key points; (c) Optimized mapping rules)

图11 抓取姿态映射流程图

Fig. 11 The flowchart of grasp mapping

表1 工具规格尺寸

Table 1 Tool size

名称	规格/mm
锤子	204×117×23
锤子2	332×130×33
电钻	203×182×57
电钻2	224×174×90

表2 实验环境配置

Table 2 Experimental environment configuration

配置项	型号
编程语言	Python3.8
深度学习框架	Pytorch2.0
操作系统	Ubuntu22.04
CPU	Intel(R) Core(TM) i9-10980XE
运行内存	128 G
GPU	NVIDIA GeForce RTX 3090

表3 评价指标

Table 3 Evaluation indicators

指标名称	指标评估内容	评估方法
手-物互穿体积	评估物理合理性	通过将网格体素化为1 mm³立方体并计算手表面内部体素体积的总和来作为互穿体积
仿真位移	评估抓取的稳定性	将物体和预测的抓取放入模拟器中，并测量物体质心在重力的影响下的平均模拟位移
手部指间自碰撞	评估手部不同区域碰撞情况	将手部的三角面片模型划分为6个区域，并将存在连接关系的区域之间的面片进行排除，避免计算碰撞关系时存在歧义，如图12所示，通过计算不同区域之间是否存在穿透来判断是否发生碰撞。计算碰撞的抓取姿态在所生成的n个抓取姿态中所占的百分比
平均最大穿透深度	评估灵巧手的抓取质量	选取n个抓取姿态，计算映射后的灵巧手与工具的凸包碰撞体之间的平均最大穿透深度
收敛比例	评估映射规则的收敛性	选取n个抓取姿态，统计在m次迭代之内，映射函数小于阈值的比例
抓取姿态的合理性	定性评估抓取姿态	以训练源数据中不同意图下的抓取姿态为参考，判断生成的抓取姿态是否符合指定的意图，抓取位置是否合适并满足视觉合理性

图12 手部碰撞区域划分

Fig. 12 Division of hand collision regions

表4 针对锤子的抓取姿态生成算法比较

Table 4 Comparison of grasp pose generation algorithms for power drill

模型	意图	穿透体积/cm³	仿真位移/m	手部自碰撞概率/%
GraspTTA	Use	1.235	0.012	0
GraspTTA	Pass	1.150	0.011	0
IntGen	Use	0.741	0.011	0
IntGen	Pass	0.542	0.021	0
IntContact	Use	0.654	0.009	12
IntContact	Pass	0.398	0.016	40

表5 针对电钻的抓取姿态生成算法比较

Table 5 Comparison of grasp pose generation algorithms for power drill

模型	意图	穿透体积/cm³	仿真位移/m	手部自碰撞概率/%
GraspTTA	Use	2.692	0.017	100
GraspTTA	Pass	2.054	0.011	0
IntGen	Use	4.732	0.029	100
IntGen	Pass	1.407	0.022	0
IntContact	Use	1.865	0.019	0
IntContact	Pass	0.719	0.012	13

图13 不同算法在不同意图下对锤子和电钻抓取姿态生成示例

Fig. 13 Examples of different algorithms for hammer and drill grasp pose generation under different intents ((a1, a2) GrasspTTA_use; (b1, b2) IntGen_use; (c1, c2) IntContact_use; (d1, d2) GraspTTA_pass; (e1, e2) IntGen_pass; (f1, f2) IntContact_pass)

表6 锤子2抓取姿态生成消融实验

Table 6 Hammer_2 grasp posture generation in ablation experiment

模型	意图	穿透体积/cm³	仿真位移/m	手部指间自碰撞/%
IntContact	Use	0.900	0.015	58
IntContact	Pass	1.171	0.028	57
IntContact+Tink	Use	0.170	0.014	35
IntContact+Tink	Pass	0.280	0.023	55
IntContact+Tink+CollisionOurs	Use	0.169	0.013	4
IntContact+Tink+CollisionOurs	Pass	0.283	0.025	10

表7 电钻2抓取姿态生成消融实验

Table 7 Power drill_2 grasp posture generation in ablation experiment

模型	意图	穿透体积/cm³	仿真位移/m	手部指间自碰撞/%
IntContact	Use	2.699	0.020	37
IntContact	Pass	2.836	0.021	63
IntContact+Tink	Use	2.198	0.012	25
IntContact+Tink	Pass	1.192	0.011	12
IntContact+Tink+CollisionOurs	Use	2.271	0.014	0
IntContact+Tink+CollisionOurs	Pass	1.212	0.011	4

图14 消融实验中锤子2和电钻2抓取姿态示例

Fig. 14 Example of grasp pose of hammer_2 and drill_2 in ablation experiments ((a1, a2) IntContact_use; (b1, b2) IntContact+Tink_use; (c1, c2) Ours_use; (d1, d2) IntContact_pass; (e1, e2) IntContact+Tink_pass; (f1, f2) Ours_pass)

表8 不同映射规则下灵巧手抓取锤子2

Table 8 Dexterous hand grasp hammer_2 under different mapping rules

映射规则	意图	平均最大穿透深度/mm	收敛比例/%
指尖	Use	11.6	10
指尖	Pass	17.8	52
指尖与近侧指间关键点	Use	5.4	76
指尖与近侧指间关键点	Pass	10.8	94
优化	Use	5.3	90
优化	Pass	9.9	94

表9 不同映射规则下灵巧手抓取电钻2

Table 9 Dexterous hand grasp drill_2 under different mapping rules

映射规则	意图	平均最大穿透深度/mm	收敛比例/%
指尖	Use	29.2	0
指尖	Pass	16.6	84
指尖与近侧指间关键点	Use	4.2	100
指尖与近侧指间关键点	Pass	13.9	86
优化	Use	3.1	100
优化	Pass	12.4	90

图15 不同映射规则下灵巧手抓取锤子2和电钻2示例

Fig. 15 Example of dexterous hand grasp hammer_2 and drill_2 under different mapping rules ((a1, a2) A_use; (b1, b2) B_use; (c1, c2) C_use; (d1, d2) A_pass; (e1, e2) B_pass; (f1, f2) C_pass)

图16 映射规则c下不同品牌灵巧手抓取锤子2，电钻2示例

Fig. 16 Example of different brands of dexterous hands grasp hammer_2, drill_2 under mapping rule c ((a1, a2) Schunk_use; (b1, b2) Shadow_use; (c1, c2) Ability_use; (d1, d2) Schunk_pass; (e1, e2) Shadow_pass; (f1, f2) Ability_pass)

图17 灵巧手机器人抓取平台

Fig. 17 Dexterous hand robot grasp platform

图18 不同意图下真机抓取示例

Fig. 18 Example of real machine grasp with different intents ((a) Hammer_pass; (b) Hammer_use; (c) Power_pass; (d) Power_use)

参考文献 21

[1]	李泳耀, 江磊, 刘宇飞, 等. 仿人灵巧手的稳定抓取方法研究综述[J]. 兵工学报, 2023, 44(11): 3237-3252. DOI
	LI Y Y, JIANG L, LIU Y F, et al. A review of stable grasping methods for humanoid dexterous hands[J]. Acta Armamentarii, 2023, 44(11): 3237-3252 (in Chinese). DOI
[2]	徐昱琳, 徐粟轩, 徐逍, 等. SHU-II五指仿人灵巧手的运动学及抓取分析[J]. 仪器仪表学报, 2018, 39(9): 30-39.
	XU Y L, XU S X, XU X, et al. Kinematics and grasping analysis of SHU-II five fingers humanoid dexterous hand[J]. Chinese Journal of Scientific Instrument, 2018, 39(9): 30-39 (in Chinese).
[3]	童立靖, 李嘉伟. 一种基于改进PointNet++网络的三维手姿估计方法[J]. 图学学报, 2022, 43(5): 892-900.
	TONG L J, LI J W. A 3D hand pose estimation method based on improved PointNet++[J]. Journal of Graphics, 2022, 43(5): 892-900 (in Chinese).
[4]	蔡世波, 陶志成, 万伟伟, 等. 机器人多指灵巧手的研究现状、趋势与挑战[J]. 机械工程学报, 2021, 57(15): 1-14. DOI
	CAI S B, TAO Z C, WAN W W, et al. Multi-fingered dexterous hands: from simplicity to complexity and simplifying complex applications[J]. Journal of Mechanical Engineering, 2021, 57(15): 1-14 (in Chinese). DOI
[5]	MILLER A T, ALLEN P K. Graspit! a versatile simulator for robotic grasping[J]. IEEE Robotics & Automation Magazine, 2004, 11(4): 110-122.
[6]	DZIDEK B M, ADAMS M J, ANDREWS J W, et al. Contact mechanics of the human finger pad under compressive loads[J]. Journal of the Royal Society Interface, 2017, 14(127): 20160935.
[7]	伍一鹤, 张振宁, 仇栋, 等. 基于深度强化学习的虚拟手自适应抓取研究[J]. 图学学报, 2021, 42(3): 462-469.
	WU Y H, ZHANG Z N, QIU D, et al. Research on adaptive grasping of virtual hands based on deep reinforcement learning[J]. Journal of Graphics, 2021, 42(3): 462-469 (in Chinese).
[8]	ZHU T Q, WU R N, LIN X B, et al. Toward human-like grasp: dexterous grasping via semantic representation of object-hand[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 15721-15731.
[9]	LIU S W, ZHOU Y, YANG J M, et al. ContactGen: generative contact modeling for grasp generation[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 20552-20563.
[10]	YANG L X, LI K L, ZHAN X Y, et al. OakInk: a large-scale knowledge repository for understanding hand-object interaction[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 20921-20930.
[11]	BRAHMBHATT S, TANG C C, TWIGG C D, et al. ContactPose: a dataset of grasps with object contact and hand pose[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 361-378.
[12]	TAHERI O, GHORBANI N, BLACK M J, et al. GRAB: a dataset of whole-body human grasping of objects[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 581-600.
[13]	SOHN K, YAN X C, LEE H. Learning structured output representation using deep conditional generative models[C]// The 28th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2015: 3483-3491.
[14]	QI C R, YI L, SU H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[C]// The 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 5105-5114.
[15]	PARK J J, FLORENCE P, STRAUB J, et al. DeepSDF: learning continuous signed distance functions for shape representation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 165-174.
[16]	LORENSEN W E, CLINE H E. Marching cubes: a high resolution 3D surface construction algorithm[M]// WOLFER. Seminal Graphics:Pioneering Efforts that Shaped the Field. New York: Association for Computing Machinery, 1998: 347-353.
[17]	ROMERO J, TZIONAS D, BLACK M J. Embodied hands: modeling and capturing hands and bodies together[J]. ACM Transactions on Graphics, 2017, 36(6): 245.
[18]	XIANG F B, QIN Y Z, MO K C, et al. SAPIEN: a simulated part-based interactive environment[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 11094-11104.
[19]	HANDA A, VAN WYK K, YANG W, et al. DexPilot: vision-based teleoperation of dexterous robotic hand-arm system[C]// 2020 IEEE International Conference on Robotics and Automation. New York: IEEE Press, 2020: 9164-9170.
[20]	QIN Y Z, WU Y H, LIU S W, et al. DexMV: imitation learning for dexterous manipulation from human videos[C]// The 17th European Conference on Computer Vision. Cham: Springer, 2022: 570-587.
[21]	JIANG H W, LIU S W, WANG J S, et al. Hand-object contact consistency reasoning for human grasps generation[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 11087-11096.

融合知识迁移的灵巧手抓取姿态生成

Grasp pose generation for dexterous hand with integrated knowledge transfer

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 27

参考文献 21

相关文章 15

编辑推荐

Metrics

本文评价

[1]	陈治彰, 封颖超杰, 翁罗轩, 沈健, 陈为. DRec：大语言模型驱动的数据分析推荐系统[J]. 图学学报, 2025, 46(5): 1028-1041.
[2]	岳子佳, 王文嵩, 陈双敏, 辛士庆, 屠长河. 跨越开边界的测地距离传播[J]. 图学学报, 2025, 46(5): 1042-1049.
[3]	左屿琪, 张云峰, 张秋悦, 徐英城. 基于超图表示学习和Transformer模型优化的知识感知推荐[J]. 图学学报, 2025, 46(5): 1050-1060.
[4]	吴浩宇, 杨小超, 王伟, 赵罡. 基于运动学原理的复合材料编织成型工艺仿真技术研究[J]. 图学学报, 2025, 46(5): 1061-1071.
[5]	扈慧强, 贺长雁, 刘小军, 贾金原, 高路. 基于Weisfeiler-Lehman核增强的无监督BIM构件检索方法[J]. 图学学报, 2025, 46(5): 1123-1133.
[6]	黄凯奇, 武美奇, 陈宏昊, 丰效坤, 张岱凌. 视觉图灵三境界：大模型时代下视觉智能进展与展望[J]. 图学学报, 2025, 46(5): 919-930.
[7]	黄敬, 时瑞浩, 宋文明, 郭和攀, 魏璜, 魏小松, 姚剑. 自动驾驶图像合成方法综述：从模拟器到新范式[J]. 图学学报, 2025, 46(5): 931-949.
[8]	翟永杰, 翟邦朝, 胡哲东, 杨珂, 王乾铭, 赵晓瑜. 基于自适应特征融合金字塔与注意力机制的输电线路绝缘子缺陷检测方法[J]. 图学学报, 2025, 46(5): 950-959.
[9]	冷烁, 王玮, 欧家勇, 薛志刚, 宋英龙, 莫斯钧. 基于大型视觉语言模型的施工现场安全监控研究[J]. 图学学报, 2025, 46(5): 960-968.
[10]	刘成, 张家意, 袁烽, 张睿, 高欣. 基于SAM2的腹腔镜手术多目标自动分割方法[J]. 图学学报, 2025, 46(5): 969-979.
[11]	叶文龙, 陈斌. PanoLoRA：基于Stable Diffusion的全景图像生成的高效微调方法[J]. 图学学报, 2025, 46(5): 980-989.
[12]	朱泓淼, 钟国杰, 张严辞. 基于均值漂移与深度学习融合的小语义点云语义分割[J]. 图学学报, 2025, 46(5): 998-1009.
[13]	郭瑞东, 蓝贵文, 范冬林, 钟展, 徐梓睿, 任新月. 基于特征聚焦扩散网络的电力巡检目标检测算法[J]. 图学学报, 2025, 46(4): 719-726.
[14]	雷松林, 赵征鹏, 阳秋霞, 普园媛, 谷金晶, 徐丹. 基于可解耦扩散模型的零样本风格迁移[J]. 图学学报, 2025, 46(4): 727-738.
[15]	陈东, 李昌隆, 杜振龙, 宋爽, 李晓丽. 光影智绘：基于SAM的视频阴影鲁棒抽取[J]. 图学学报, 2025, 46(4): 739-745.