基于双语义双向对齐VAE的广义零样本学习

doi:10.11996/JG.j.2095-302X.2023030521

图学学报 ›› 2023, Vol. 44 ›› Issue (3): 521-530.DOI: 10.11996/JG.j.2095-302X.2023030521

• 图像处理与计算机视觉 • 上一篇下一篇

基于双语义双向对齐VAE的广义零样本学习

史彩娟¹^,²(), 石泽¹^,², 闫巾玮¹^,², 毕阳阳¹^,²

1.华北理工大学人工智能学院，河北唐山 063210
2.河北省工业智能感知重点实验室，河北唐山 063210

收稿日期:2022-09-26 接受日期:2022-12-06 出版日期:2023-06-30 发布日期:2023-06-30
作者简介:
史彩娟(1977-)，女，教授，博士。主要研究方向为计算机视觉、图像处理和深度学习。E-mail：scj-blue@163.com
基金资助:
华北理工大学杰出青年基金项目(JQ201715);唐山市人才项目(A202110011)

Bi-directionally aligned VAE based on double semantics for generalized zero-shot learning

SHI Cai-juan¹^,²(), SHI Ze¹^,², YAN Jin-wei¹^,², BI Yang-yang¹^,²

1. College of Artificial Intelligence, North China University of Science and Technology, Tangshan Hebei 063210, China
2. Hebei Key Laboratory of Industrial Intelligent Perception, Tangshan Hebei 063210, China

Received:2022-09-26 Accepted:2022-12-06 Online:2023-06-30 Published:2023-06-30
About author:
SHI Cai-juan (1977-), professor, Ph.D. Her main research interests cover computer vision, image processing and deep learning. E-mail：scj-blue@163.com
Supported by:
Distinguished Youth Foundation of North China University of Science and Technology(JQ201715);Talent Foundation of Tangshan(A202110011)

摘要/Abstract

摘要：

广义零样本学习(GZSL)旨在利用视觉特征和语义信息之间的关系来同时识别可见类和不可见类。现有的大部分方法使用生成模型生成不可见类的伪视觉特征，但一般采用单向对齐VAE且语义原型种类单一，导致不可见类的语义信息非常有限。因此，提出了一种基于双语义双向对齐变分自编码器的广义零样本学习模型，首先采用户定义的属性和词向量两种语义原型，基于双向对齐的VAE分别稳定地生成2种伪视觉特征来获取丰富的语义信息；然后，设计了特征融合模块对2种伪视觉特征进行有效融合，并去除其中的冗余信息，增强伪视觉特征表示；最后，采用分类正则化进一步增强伪视觉特征的类别独立性。在3个基准数据集上进行了大量实验，并与相关算法模型进行了比较，结果表明了该模型的有效性。

关键词: 广义零样本学习, 生成模型, 双语义原型, 双向对齐变分自编码器, 特征融合增强

Abstract:

Generalized zero-shot learning (GZSL) aims to recognize both seen and unseen classes by utilizing the relationship between visual features and semantic information. However, existing GZSL methods mostly rely on generative models to generate pseudo visual features for unseen classes. The problem with these models is that they commonly employ unidirectional VAE and a single type of semantic prototype, which limits the obtained semantic information of unseen classes. To address this issue, a bi-directionally aligned VAE based on a double semantics model (BAVAE-DS) for GZSL was proposed. First, two types of prototypes, i.e., user-defined attributes and word vectors, were adopted to steadily generate two types of pseudo visual features respectively using the bi-directionally aligned VAE. This resulted in abundant semantic information that could be used to represent unseen classes. Next, a feature fusion model was designed to fuse the two types of pseudo visual features and remove the redundancy, thus enhancing the pseudo visual features. Finally, classification regularization was employed to enhance the independence of classes in the classification module. Extensive experiments were conducted on three benchmark datasets and the results were compared with other methods, proving the effectiveness of the proposed model.

Key words: generalized zero-shot learning, generative model, double semantic prototypes, bi-directionally aligned VAE, feature fusion and enhancement

中图分类号:

TP391

史彩娟, 石泽, 闫巾玮, 毕阳阳. 基于双语义双向对齐VAE的广义零样本学习[J]. 图学学报, 2023, 44(3): 521-530.

SHI Cai-juan, SHI Ze, YAN Jin-wei, BI Yang-yang. Bi-directionally aligned VAE based on double semantics for generalized zero-shot learning[J]. Journal of Graphics, 2023, 44(3): 521-530.

图/表 8

图1 BAVAE-DS框架图

Fig. 1 The framework of BAVAE-DS

表1 在3个基准数据集上广义零样本学习性能比较(%)

Table 1 Performance comparison of GZSL on three benchmark datasets (%)

模型		AwA1			AwA2			CUB
模型		U	S	H	U	S	H	U	S	H
基于空间嵌入方法	DAP^[32]	0.0	88.7	0.0	0.0	84.7	0.0	1.7	67.9	3.3
	ESZSL^[4]	6.6	75.6	12.1	5.9	77.8	11.0	12.6	63.8	21.0
	LATEM^[26]	7.3	71.7	13.3	11.5	77.3	20.0	15.2	57.3	24.0
	SJE^[25]	11.3	74.6	19.6	8.0	73.9	14.4	23.5	59.2	33.6
基于生成模型方法	f-CLSWGAN^[15]	57.9	61.4	59.6	52.1	68.9	59.4	43.7	57.7	49.7
	SE^[21]	56.6	67.8	61.5	58.9	68.1	62.8	41.5	53.3	46.7
	M2GAN^[13]	42.1	80.7	54.7	38.3	85.0	52.8	20.7	58.2	30.5
	MGM-VAE^[33]	-	-	-	61.4	67.8	64.5	49.9	57.9	53.6
	LVA^[34]	54.4	70.0	61.2	-	-	-	45.6	53.2	48.5
	SMF-VAE^[35]	-	-	63.8	-	-	-	-	-	52.3
	CADA-VAE^[11]	57.3	72.8	64.1	55.8	75.0	63.9	51.6	53.5	52.4
本文	BAVAE-DS	59.5	74.6	66.2	56.9	77.2	65.5	52.9	57.5	55.1

图2 伪视觉特征生成模块性能分析

Fig. 2 Performance analysis of pseudo visual feature generation module

图3 伪视觉特征生成模块t-SNE可视化((a)单语义单向对齐SE；(b)单语义双向对齐CADA-VAE；(c)双语义双向对齐BAVAE-DS)

Fig. 3 T-SNE visualization analysis of pseudo visual feature generation module ((a) SE based on single semantic and unidirectional VAE; (b) CADA-VAE based on single semantic and bi-directional VAE; (c) BAVAE-DS based on double semantic and bi-directional VAE)

图4 特征融合模块的权重分析((a) U的性能；(b) S的性能；(c) H的性能)

Fig. 4 Weight analysis of feature fusion module ((a) Performance of U; (b) Performance of S; (c) Performance of H)

图5 分析超参数α ((a) U的性能；(b) S的性能；(c) H的性能)

Fig. 5 Analysis of hyperparameter α ((a) Performance of U; (b) Performance of S; (c) Performance of H)

图6 分析超参数γ ((a) U的性能；(b) S的性能；(c) H的性能)

Fig. 6 Analysis of hyperparameter γ ((a) Performance of U; (b) Performance of S; (c) Performance of H)

表2 在3个基准数据集上的消融实验

Table 2 Ablation study on three benchmark datasets

模型	损失函数	AwA1 (%)	AwA2 (%)	CUB (%)
BAVAE-DS#1	L₁	64.1	63.9	52.4
BAVAE-DS#2	L₂	58.4	55.7	50.9
BAVAE-DS	L₁L₂L₃	66.2	65.5	55.1

参考文献 35

[1]	LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436-444. DOI
[2]	LAMPERT C H, NICKISCH H, HARMELING S. Learning to detect unseen object classes by between-class attribute transfer[C]// 2009 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2009: 951-958.
[3]	芦楠楠, 刘一雄, 邱铭恺. 基于随机传播图卷积模型的零样本图像分类[J]. 图学学报, 2022, 43(4): 624-632.
	LU N N, LIU Y X, QIU M K. Zero-shot image classification based on random propagation graph convolution model[J]. Journal of Graphics, 2022, 43(4): 624-632. (in Chinese)
[4]	ROMERA-PAREDES B, TORR P H. An embarrassingly simple approach to zero-shot learning[C]// The 32nd International Conference on Machine Learning. New York: ACM, 2015: 2152-2161.
[5]	LI X. Learning unseen visual prototypes for zero-shot classification[J]. Knowledge-Based Systems, 2018, 160: 176-187. DOI URL
[6]	XIAN Y Q, SCHIELE B, AKATA Z. Zero-shot learning—the good, the bad and the ugly[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 3077-3086.
[7]	DING J Y, HU X, ZHONG X R. A semantic encoding out-of-distribution classifier for generalized zero-shot learning[J]. IEEE Signal Processing Letters, 2021, 28: 1395-1399. DOI URL
[8]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]// The 27th International Conference on Neural Information Processing Systems. New York: ACM, 2014, 2: 2672-2680.
[9]	KINGMA D P, WELLING M. Auto-encoding variational Bayes[EB/OL]. [2022-03-05]. https://arxiv.org/abs/1312.6114.
[10]	XIAN Y Q, SHARMA S, SCHIELE B, et al. F-VAEGAN-D2: a feature generating framework for any-shot learning[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 10267-10276.
[11]	SCHÖNFELD E, EBRAHIMI S, SINHA S, et al. Generalized zero- and few-shot learning via aligned variational autoencoders[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 8239-8247.
[12]	FELIX R, VIJAY KUMAR B G, REID I, et al. Multi-modal cycle-consistent generalized zero-shot learning[M]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 21-37.
[13]	JI Z. Multi-modal generative adversarial network for zero-shot learning[J]. Knowledge-Based Systems, 2020, 197: 105847. DOI URL
[14]	KOBYZEV I, PRINCE S J D, BRUBAKER M A. Normalizing flows: an introduction and review of current methods[EB/OL]. [2022-03-05]. https://arxiv.org/abs/1908.09257.
[15]	XIAN Y Q, LORENZ T, SCHIELE B, et al. Feature generating networks for zero-shot learning[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 5542-5551.
[16]	VERMA V K, BRAHMA D, RAI P. Meta-learning for generalized zero-shot learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(4): 6062-6069. DOI URL
[17]	LIU H. Dual-stream generative adversarial networks for distributionally robust zero-shot learning[J]. Information Sciences, 2020, 519: 407-422. DOI URL
[18]	GAO R, HOU X S, QIN J, et al. Zero-VAE-GAN: generating unseen features for generalized and transductive zero-shot learning[J]. IEEE Transactions on Image Processing, 2020, 29: 3665-3680. DOI URL
[19]	ZHANG Z L, LI Y J, YANG J, et al. Cross-layer autoencoder for zero-shot learning[J]. IEEE Access, 2019, 7: 167584-167592. DOI
[20]	MISHRA A, REDDY M S K, MITTAL A, et al. A generative model for zero shot learning using conditional variational autoencoders[EB/OL]. [2022-03-05]. https://arxiv.org/abs/1709.00663.
[21]	VERMA V K, ARORA G, MISHRA A, et al. Generalized zero-shot learning via synthesized examples[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 4281-4289.
[22]	YU H, LEE B. Zero-shot learning via simultaneous generating and learning[EB/OL]. [2022-03-05]. https://arxiv.org/abs/1910.09446.
[23]	MA P R, HU X. A variational autoencoder with deep embedding model for generalized zero-shot learning[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 11733-11740. DOI URL
[24]	GU Y C, ZHANG L, LIU Y, et al. Generalized zero-shot learning via VAE-conditioned generative flow[EB/OL]. [2022-03-05]. https://arxiv.org/abs/2009.00303.
[25]	AKATA Z, REED S, WALTER D, et al. Evaluation of output embeddings for fine-grained image classification[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 2927-2936.
[26]	XIAN Y Q, AKATA Z, SHARMA G, et al. Latent embeddings for zero-shot classification[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 69-77.
[27]	XIE C. Cross knowledge-based generative zero-shot learning approach with taxonomy regularization[J]. Neural Networks, 2021, 139: 168-178. DOI PMID
[28]	XIANG H X, XIE C, ZENG T, et al. Multi-knowledge fusion for new feature generation in generalized zero-shot learning[EB/OL]. [2022-03-05]. https://arxiv.org/abs/2102.11566.
[29]	MALL U, HARIHARAN B, BALA K. Zero-shot learning using multimodal descriptions[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2022: 3930-3938.
[30]	MERCEA O B, RIESCH L, KOEPKE A S, et al. Audiovisual generalised zero-shot learning with cross-modal attention and language[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 10543-10553.
[31]	WAH C, BRANSON S, WELINDER P, et al. The caltech-UCSD Birds-200-2011 Dataset[EB/OL]. [2022-03-05]. https://www.researchgate.net/publication/251734721_The_Caltech-UCSD_Birds200-2011_Dataset.
[32]	LAMPERT C H, NICKISCH H, HARMELING S. Attribute-based classification for zero-shot visual object categorization[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(3): 453-465. DOI PMID
[33]	邵洁, 李晓瑞. 基于混合高斯分布的广义零样本识别[J]. 上海电力大学学报, 2021, 37(5): 475-480.
	SHAO J, LI X R. A method for generalized zero-shot learning based on Gaussian mixture distribution[J]. Journal of Shanghai University of Electric Power, 2021, 37(5): 475-480. (in Chinese)
[34]	钟小容, 胡晓, 丁嘉昱. 基于潜层向量对齐的持续零样本学习算法[J]. 模式识别与人工智能, 2021, 34(12): 1152-1159. DOI
	ZHONG X R, HU X, DING J Y. Continual zero-shot learning algorithm based on latent vectors alignment[J]. Pattern Recognition and Artificial Intelligence, 2021, 34(12): 1152-1159. (in Chinese) DOI
[35]	林爽, 王晓军. 运用模态融合的半监督广义零样本学习[J]. 计算机工程与应用, 2022, 58(5): 163-171. DOI
	LIN S, WANG X J. Semi-supervised generalized zero-shot learning using modal fusion[J]. Computer Engineering and Applications, 2022, 58(5): 163-171. (in Chinese) DOI

基于双语义双向对齐VAE的广义零样本学习

Bi-directionally aligned VAE based on double semantics for generalized zero-shot learning

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 35

相关文章 1

编辑推荐

Metrics

本文评价