Hierarchical alignment for zero-shot sketch-based 3D shape retrieval

doi:10.11996/JG.j.2095-302X.2026040844

Abstract

Abstract:

Sketch-based 3D shape retrieval has gradually become an important research direction in the field of 3D shape retrieval, mainly because it provides an intuitive mode of human-computer interaction for users. However, in real application scenarios, the number of 3D shape categories is very large and continues to grow, which makes it hard for training data to cover all potential categories, and thus traditional supervised retrieval methods based on category labels often struggle to meet the demand for new-category retrieval in open environments. This limitation motivates the study of zero-shot sketch-based 3D shape retrieval, which aims to improve the adaptability of retrieval systems to open scenarios. In this task, there are two main difficulties: addressing the modality gap between sketches and 3D shapes, and achieving effective knowledge transfer from seen categories to unseen categories in the absence of samples from unseen classes. To address these problems, a hierarchical alignment method was designed, which performs progressive alignment at both the input level and the feature level to jointly reduce cross-modal differences and support knowledge transfer. At the input level, an adversarial learning-based pseudo-view generation method was introduced to convert sketches into pseudo multi-view representations that are closer to 3D shape distributions, thereby reducing modality discrepancy from the source. At the feature level, a shared semantic embedding space was constructed, and a multi-granularity metric learning strategy was further introduced, in which semantic consistency constraints, semantic enhancement mechanisms, and class-level proxy constraints were jointly used to improve feature representation. Experimental results on the SHREC 2013 and SHREC 2014 datasets showed that the proposed hierarchical alignment framework achieved competitive performance across multiple evaluation metrics, while ablation studies further verified the effectiveness of each component, indicating that input-level alignment and feature-level semantic alignment can work together to enhance cross-modal representation learning and improve zero-shot retrieval performance.

Key words: sketch-based 3D shape retrieval, zero-shot learning, deep learning, adversarial generation, cross-modal alignment

CLC Number:

TP391

WANG Yutao, YANG Chao, KUANG Liqun, YANG Xiaowen, HAN Xie, JIAO Shichao. Hierarchical alignment for zero-shot sketch-based 3D shape retrieval[J]. Journal of Graphics, 2026, 47(4): 844-853.

Figures/Tables 13

References 41

[1]	DAI Y, FENG Y F, MA N, et al. Cross-modal 3D shape retrieval via heterogeneous dynamic graph representation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, 47(4): 2370-2387. DOI PMID
[2]	SU Y W, LI W J, BAI J, et al. SKD-SBSR: structural knowledge distillation for sketch-based 3D shape retrieval[J]. Knowledge-Based Systems, 2025, 310: 112891. DOI URL
[3]	SU Y W, BAI J, LIN G. DKD²L: dual knowledge distillation dynamic learning for sketch-based 3D shape retrieval[C]// ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing. New York: IEEE Press, 2025: 1-5.
[4]	YUAN S H, WEN C C, LIU Y S, et al. Retrieval-specific view learning for sketch-to-shape retrieval[J]. IEEE Transactions on Multimedia, 2025, 27: 768-779. DOI URL
[5]	WANG B R, ZHOU Y. Doodle to object: practical zero-shot sketch-based 3D shape retrieval[C]// The 37th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2023: 2474-2482.
[6]	XU R, HAN Z Y, HUI L, et al. Domain disentangled generative adversarial network for zero-shot sketch-based 3D shape retrieval[C]// The 36th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2022: 2902-2910.
[7]	MENG M, CHEN W H, LIU J G, et al. CoDi: contrastive disentanglement generative adversarial networks for zero-shot sketch-based 3D shape retrieval[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2025, 35(2): 1910-1920. DOI URL
[8]	POURPANAH F, ABDAR M, LUO Y X, et al. A review of generalized zero-shot learning methods[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(4): 4051-4070.
[9]	ZHU C J, CUI D D, JIA Q, et al. Sketch-based 3D shape retrieval with multi-view fusion transformer[C]// ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing. New York: IEEE Press, 2024: 3005-3009.
[10]	唐静, 彭伟龙, 唐可可, 等. 基于多视图网络三维形状检索的通用扰动攻击[J]. 图学学报, 2022, 43(1): 93-100.
	TANG J, PENG W L, TANG K K, et al. MvUPA: universal perturbation attack against 3D shape retrieval based on multi-view networks[J]. Journal of Graphics, 2022, 43(1): 93-100 (in Chinese).
[11]	WANG F, KANG L, LI Y. Sketch-based 3D shape retrieval using convolutional neural networks[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 1875-1883.
[12]	ZHU F, XIE J, FANG Y. Learning cross-domain neural networks for sketch-based 3D shape retrieval[C]// The 30th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2016: 3683-3689.
[13]	CHEN J X, FANG Y. Deep cross-modality adaptation via semantics preserving adversarial learning for sketch-based 3D shape retrieval[C]// The 15th European Conference on Computer Vision. Cham: Springer, 2018: 624-640.
[14]	HE X W, ZHOU Y, ZHOU Z C, et al. Triplet-center loss for multi-view 3D object retrieval[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 1945-1954.
[15]	杨瞻源, 白静, 李文静, 等. 面向三维模型草图检索的三元层次度量网络[J]. 计算机辅助设计与图形学学报, 2024, 36(11): 1791-1804.
	YANG Z Y, BAI J, LI W J, et al. Triplet hierarchical metric network for sketch-based 3D shape retrieval[J]. Journal of Computer-Aided Design & Computer Graphics, 2024, 36(11): 1791-1804 (in Chinese).
[16]	LIANG S, DAI W D, CHENG C M, et al. Rethinking sketch- based 3D shape retrieval: a simple baseline and benchmark reconstruction[J]. Neurocomputing, 2025, 618: 128990. DOI URL
[17]	BAI S J, BAI J. HDA²L: hierarchical domain-augmented adaptive learning for sketch-based 3D shape retrieval[J]. Knowledge-Based Systems, 2023, 264: 110302. DOI URL
[18]	LI W J, BAI J, ZHENG H. D²GL: dual-level dual-scale graph learning for sketch-based 3D shape retrieval[J]. Pattern Recognition, 2024, 156: 110768. DOI URL
[19]	BAI S J, BAI J, XU H, et al. PAGML: precise alignment guided metric learning for sketch-based 3D shape retrieval[J]. Image and Vision Computing, 2023, 136: 104756. DOI URL
[20]	LIANG S, DAI W D, WEI Y C. Uncertainty learning for noise resistant sketch-based 3D shape retrieval[J]. IEEE Transactions on Image Processing, 2021, 30: 8632-8643. DOI URL
[21]	HOU W T, DIAO Z Y, PENG J L. Sketch-based 3D shape retrieval via cross-modal contrastive learning and difficulty-aware uncertainty regularization[C]// The 7th Chinese Conference on Pattern Recognition and Computer Vision. Cham: Springer: 2024: 521-534.
[22]	BAI S J, LI Y L, CHANG R H, et al. SCDL: sketch causal disentangled learning for sketch-based 3D shape retrieval[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2025, 35(7): 7239-7252. DOI URL
[23]	白静, 袁涛, 范有福. ZS3D-Net: 面向三维模型的零样本分类网络[J]. 计算机辅助设计与图形学学报, 2022, 34(7): 1118-1126.
	BAI J, YUAN T, FAN Y F. ZS3D-Net: zero-shot classification network for 3D models[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(7): 1118-1126 (in Chinese).
[24]	SU Y T, LI J Y, LI W H, et al. Semantically guided projection for zero-shot 3D model classification and retrieval[J]. Multimedia Systems, 2022, 28(6): 2437-2451. DOI
[25]	晏浩, 白静, 郑虎. 一致性约束引导的零样本三维模型分类网络[J]. 中国图象图形学报, 2025, 30(5): 1450-1465.
	YAN H, BAI J, ZHENG H. Consistency constraint guided network for zero-shot 3D classification[J]. Journal of Image and Graphics, 2025, 30(5): 1450-1465 (in Chinese). DOI URL
[26]	范有福, 白静, 邵会会, 等. 判别性特征引导的零样本三维模型分类算法[J]. 计算机辅助设计与图形学学报, 2024, 36(2): 223-235.
	FAN Y F, BAI J, SHAO H H, et al. Discriminative feature-guided zero-shot learning of 3D model classification algorithm[J]. Journal of Computer-Aided Design & Computer Graphics, 2024, 36(2): 223-235 (in Chinese).
[27]	HAO Y, SU Y K, LIN G S, et al. Contrastive generative network with recursive-loop for 3D point cloud generalized zero-shot classification[J]. Pattern Recognition, 2023, 144: 109843. DOI URL
[28]	XU Y, FENG Y F, JIANG Y. Structure-aware residual-center representation for self-supervised open-set 3D cross-modal retrieval[C]// 2024 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2024: 1-6.
[29]	XU Y, FENG Y F, ZHUANG X, et al. Residual fuzzy alignment on hypergraph for open-set 3D cross-modal retrieval[J]. IEEE Transactions on Multimedia, 2025, 27: 7285-7298. DOI URL
[30]	FENG Y F, JI S Y, LIU Y S, et al. Hypergraph-based multi-modal representation for open-set 3D object retrieval[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 46(4): 2206-2223. DOI URL
[31]	PENG B, CHEN L, SONG J H, et al. ZS-SBPRnet: a zero-shot sketch-based point cloud retrieval network based on feature projection and cross-reconstruction[J]. IEEE Transactions on Industrial Informatics, 2023, 19(8): 9194-9203. DOI URL
[32]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2242-2251.
[33]	ZHAI A, WU H Y. Classification is a Strong Baseline for Deep Metric Learning[EB/OL]. [2026-01-10]. https://www.semanticscholar.org/paper/Classification-is-a-Strong-Baseline-for-Deep-Metric-Zhai-Wu/807257e0a934d3600db9d9b4a12f7194ac8ae2c2
[34]	LI B, LU Y, GODIL A, SCHRECK T, et al. SHREC’13 track: large scale sketch-based 3D shape retrieval[C]// The 6th Eurographics Workshop on 3D Object Retrieval. Goslar: Eurographics Association, 2013: 89-96.
[35]	LI B, LU Y, LI C F, et al. SHREC’14 track: extended large scale sketch-based 3D shape retrieval[C]// The 7th Eurographics Workshop on 3D Object Retrieval. Goslar: Eurographics Association, 2014: 121-130.
[36]	DAI G X, XIE J, FANG Y. Deep correlated holistic metric learning for sketch-based 3D shape retrieval[J]. IEEE Transactions on Image Processing, 2018, 27(7): 3374-3386. DOI PMID
[37]	DAI W D, LIANG S. Cross-modal guidance network for sketch-based 3D shape retrieval[C]// 2020 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2020: 1-6.
[38]	REN H, ZHENG Z Q, WU Y, et al. ACNet: approaching- and-centralizing network for zero-shot sketch-based image retrieval[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(9): 5022-5035. DOI URL
[39]	JIAO S C, HAN X, KUANG L Q, et al. Global semantics correlation transmitting and learning for sketch-based cross-domain visual retrieval[J]. Complex & Intelligent Systems, 2024, 10(5): 6931-6952.
[40]	LIU D C, LUO X, PENG C L, et al. Symmetrical bidirectional knowledge alignment for zero-shot sketch-based image retrieval[J]. Neural Networks, 2026, 193: 107980 DOI URL
[41]	陈文航. 基于草图的零样本三维模型检索研究[D]. 广州: 广东工业大学, 2025.
	CHEN W H. Research on sketch-based zero-shot 3D shape retrieval[D]. Guangzhou: Guangdong University of Technology, 2025 (in Chinese).

数据集	种类	查询数量		目标数量
数据集	种类	训练	测试	目标数量
SHREC 2013	90	4 500	2 700	1 258
SHREC 2014	171	8 550	5 130	8 987

数据集	种类	查询数量		目标数量
数据集	种类	训练	测试	目标数量
SHREC 2013	90	4 500	2 700	1 258
SHREC 2014	171	8 550	5 130	8 987

数据集分类	参考文献	SHREC 2013		SHREC 2014
数据集分类	参考文献	训练种类	测试种类	训练种类	测试种类
1	[5]	67	23	133	38
2	[6] [7]	79	11	151	20

数据集分类	参考文献	SHREC 2013		SHREC 2014
数据集分类	参考文献	训练种类	测试种类	训练种类	测试种类
1	[5]	67	23	133	38
2	[6] [7]	79	11	151	20

方法	NN	FT	ST	DCG	MAP
Siamese-CNN^[11]	0.137	0.124	0.117	0.186	0.201
DCHML^[36]	0.183	0.211	0.136	0.238	0.339
TCL^[14]	0.250	0.178	0.203	0.314	0.317
CGN^[37]	0.333	0.294	0.225	0.398	0.386
PCL^[5]	0.383	0.389	0.261	0.470	0.480
ACNet^[38]	0.455	0.354	0.266	0.594	0.432
GFCFS^[39]	0.433	0.344	0.273	0.598	0.437
SBKA^[40]	0.253	0.198	0.158	0.454	0.251
HA (Ours)	0.486	0.408	0.304	0.637	0.490