Self-supervised active label cleaning

doi:10.11996/JG.j.2095-302X.2024030495

Abstract

Abstract:

Active label cleaning utilizes the active learning method for label noise processing to lower the cost of manual annotation. However, the existing active label cleaning methods still suffer from high cost of extra manual annotation, particularly due to a high proportion of correctly labeled samples among the selected suspicious ones. To address this problem, a self-supervised active label cleaning method based on core-set was proposed. Firstly, self-supervised tasks were employed for representation learning of all samples, followed by mapping the samples to a future space. Suspicious samples were then identified using a greedy K-Center set covering method, and label noise samples were selected for re-labeling based on uncertainty. By considering both the representativeness and uncertainty of samples, this method could effectively lower the proportion of correct samples in suspicious ones. Experimental results on public datasets with varying proportions of label noise demonstrated that the proposed method could significantly reduce the cost of extra manual annotation in each iteration, while also mitigating the cold start problem to some extent. Additionally, the effectiveness of the self-supervised core-set sampling module and the uncertainty prediction module in this method were validated through ablation experiments.

Key words: active learning, self-supervised learning, label noise, label cleaning, cost of extra manual annotation

CLC Number:

TP391

LIN Xiao, ZHANG Qiuyang, ZHENG Xiaomei, YANG Qizhe. Self-supervised active label cleaning[J]. Journal of Graphics, 2024, 45(3): 495-504.

Figures/Tables 9

Fig. 1 Traditional active learning framework and active label cleaning framework

Fig. 2 Overall structure diagram of self-supervised active label cleaning

Fig. 3 Cumulative class distribution heat map of data extracted for each cycle on the CIFAR10N (50% label noise) dataset ((a) Random sampling; (b) Uniform sampling; (c) Self-supervised core set sampling)

Fig. 4 Label cleaning performance under different pretext tasks on the CIFAR10N (50% label noise) dataset

Fig. 5 Label cleaning performance on the CIFAR10N (50% label noise) dataset ((a) ResNet-18; (b) VGG19)

Fig. 6 Cumulative additional manual annotation costs on the CIFAR10N (20%, 40%, 50% label noise) dataset

Table 1 Label cleaning rates on CIFAR10N and Fashion MNIST_N datasets

数据集	方法	噪声比例/%	K值	标签正确率/%
CIFAR10N	Ours	20	1 000	96.60
CIFAR10N	ALC	20	1 000	95.84
CIFAR10N	Bernhardt	20	1 000	96.32
CIFAR10N	Ours	50	2 000	98.90
CIFAR10N	ALC	50	2 000	98.87
CIFAR10N	Bernhardt	50	2 000	98.85
Fashion MNIST_N	Ours	20	1 000	96.01
Fashion MNIST_N	ALC	20	1 000	95.98
Fashion MNIST_N	Bernhardt	20	1 000	95.64
Fashion MNIST_N	Ours	50	2 000	98.90
Fashion MNIST_N	ALC	50	2 000	98.32
Fashion MNIST_N	Bernhardt	50	2 000	98.82

Fig. 7 Part of the visual samples taken on CIFAR10N (50% label noise) ((a) Clear samples which are distinctly mislabelled; (b)~(c) Fuzzy samples which are difficult to predict and mislabelled)

Fig. 8 Precision comparison results on the CIFAR10N (50% label noise) dataset

References 23

[1]	HAN B, YAO Q M, YU X R, et al. Co-teaching: robust training of deep neural networks with extremely noisy labels[C]// The 32nd International Conference on Neural Information Processing Systems. New York: ACM, 2018: 8536-8546.
[2]	DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]// 2009 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2009: 248-255.
[3]	ZHU X Q, WU X D. Class noise vs. attribute noise: a quantitative study[J]. Artificial Intelligence Review, 2004, 22(3): 177-210.
[4]	KIM Y, YIM J, YUN J, et al. NLNL: negative learning for noisy labels[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 101-110.
[5]	TONEVA M, SORDONI A, DES COMBES R T, et al. An empirical study of example forgetting during deep neural network learning[EB/OL]. (2019-11-15) [2023-03-20]. https://arxiv.org/abs/1812.05159.pdf.
[6]	HUANG J C, QU L, JIA R F, et al. O₂U-net: a simple noisy label detection approach for deep neural networks[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 3326-3334.
[7]	REBBAPRAGADA U D. Strategic targeting of outliers for expert review[EB/OL]. (2010-06-01) [2023-03-20]. blob: https://www.proquest.com/c4d84676-07c3-4eb0-84a0-89e32ab6bcbe.
[8]	EKAMBARAM R, FEFILATYEV S, SHREVE M, et al. Active cleaning of label noise[J]. Pattern Recognition, 2016, 51: 463-480.
[9]	BERNHARDT M, CASTRO D C, TANNO R, et al. Active label cleaning for improved dataset quality under resource constraints[J]. Nature Communications, 2022, 13(1): 1161. DOI PMID
[10]	JAISWAL A, BABU A R, ZADEH M Z, et al. A survey on contrastive self-supervised learning[EB/OL]. (2021-02-07) [2023-03-20]. http://arxiv.org/abs/2011.00362.pdf.
[11]	SENER O, SAVARESE S. Active learning for convolutional neural networks: a core-set approach[EB/OL]. (2018-06-01) [2023-03-20]. http://arxiv.org/abs/1708.00489.pdf.
[12]	HOULSBY N, HUSZÁR F, GHAHRAMANI Z, et al. Bayesian active learning for classification and preference learning[EB/OL]. (2011-12-24) [2023-03-20]. http://arxiv.org/abs/1112.5745.pdf.
[13]	GAL Y, GHAHRAMANI Z. Dropout as a Bayesian approximation: representing model uncertainty in deep learning[C]// The 33rd International Conference on International Conference on Machine Learning - Volume 48. New York:ACM, 2016: 1050-1059.
[14]	KIRSCH A, VAN AMERSFOORT J, GAL Y. BatchBALD: efficient and diverse batch acquisition for deep Bayesian active learning[EB/OL]. (2019-10-28) [2023-03-20]. http://arxiv.org/abs/1906.08158.pdf.
[15]	BORSOS Z, MUTNÝ M, KRAUSE A. Coresets via bilevel optimization for continual learning and streaming[EB/OL]. (2020-10-22) [2023-03-20]. http://arxiv.org/abs/2006.03875.pdf.
[16]	PINSLER R, GORDON J, NALISNICK E, et al. Bayesian batch active learning as sparse subset approximation[EB/OL]. (2021-02-08) [2023-03-20]. http://arxiv.org/abs/1908.02144.pdf.
[17]	DOERSCH C, GUPTA A, EFROS A A. Unsupervised visual representation learning by context prediction[C]// 2015 IEEE International Conference on Computer Vision. New York: IEEE Press, 2015: 1422-1430.
[18]	ZHANG R, ISOLA P, EFROS A A. Colorful image colorization[C]// European Conference on Computer Vision. Cham: Springer, 2016: 649-666.
[19]	PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 2536-2544.
[20]	GIDARIS S, SINGH P, KOMODAKIS N. Unsupervised representation learning by predicting image rotations[EB/OL]. (2018-03-21) [2023-03-20]. http://arxiv.org/abs/1803.07728.pdf.
[21]	ALGAN G, ULUSOY I. Image classification with deep learning in the presence of noisy labels: a survey[J]. Knowledge-Based Systems, 2021, 215: 106771.
[22]	MANWANI N, SASTRY P S. Noise tolerance under risk minimization[J]. IEEE Transactions on Cybernetics, 2013, 43(3): 1146-1151. DOI PMID
[23]	KRIZHEVSKY A, HINTON G. Learning multiple layers of features from tiny images[EB/OL]. (2009-04-08) [2023-03-20]. https://www.cs.toronto.edu/-kriz/learning-features-2009-TR.pdf.