Reference based transformer texture migrates depth images super resolution reconstruction

doi:10.11996/JG.j.2095-302X.2023050861

Abstract

Abstract:

Depth images contain scene depth information and exhibit strong robustness to variations in color and lighting, making them widely used in fields such as stereo vision. However, due to the limitations in depth sensor performance and the complexity of imaging environments, it is challenging to directly obtain high-quality, high-resolution depth images. To address the problem of unclear edge details in reconstructed depth images, a reference-based Transformer texture transfer method for deep image super-resolution reconstruction was proposed. For the preprocessed low-resolution depth images (LR_D) and reference images (Ref) feature blocks, similarity calculation was performed using normalized inner product. The method integrated Transformer to calculate the confidence of similarity positions, and combined it with an attention mechanism for texture transfer. Finally, the method combined the features of the low-resolution depth images to improve image detail clarity and further accurately reconstruct the results. The experimental results demonstrated that compared to other methods, the proposed method could achieve higher structural similarity (SSIM) values, and that both subjective visual effects and objective evaluation indicators have been significantly improved, indicating the excellence of the reconstruction performance.

Key words: deep learning, super-resolution reconstruction, depth image, Transformer, attention mechanism

CLC Number:

TP391

YANG Chen-cheng, DONG Xiu-cheng, HOU Bing, ZHANG Dang-cheng, XIANG Xian-ming, FENG Qi-ming. Reference based transformer texture migrates depth images super resolution reconstruction[J]. Journal of Graphics, 2023, 44(5): 861-867.

Figures/Tables 8

Fig. 1 Block diagram of texture migration network in Transformer based on reference

Fig. 2 LR_D and Ref similarity estimation process diagram

Fig. 3 Texture feature migration process diagram

Table 1 Experimental software and hardware on figuration

名称	实验配置
操作系统	ubantu 20.04
编程语言	Python 3.7.9
深度学习框架	PyTorch 1.8.0
CPU	Intel Xeon Gold 6226
GPU	NVIDIA RTX3090
Cuda	Cuda 11.1
平台	Pycharm-2021.2.1

Fig. 4 Dataset image example ((a) HR_D image; (b) Ref image)

Fig. 5 Comparison of reconstruction effects of different algorithms on Mid3 when the up-sampling factor is 4 ((a) High resolution depth images; (b) Bicubic interpolation results; (c) SRCNN algorithm results; (d) ESPCN algorithm results; (e) Algorithm results in this article; (f) Partial enlarged view of figure (a); (g) Partial enlarged view of figure (b); (h) Partial enlarged view of figure (c); (i) Partial enlarged view of figure (d); (j) Partial enlarged view of figure (e))

Fig. 6 Comparison of reconstruction effects of different algorithms on Mid3 when the up-sampling factor is 2 ((a) High resolution depth images; (b) Bicubic interpolation results; (c) SRCNN algorithm results; (d) ESPCN algorithm results; (e) Algorithm results in this article; (f) Partial enlarged view of figure (a); (g) Partial enlarged view of figure (b); (h) Partial enlarged view of figure (c); (i) Partial enlarged view of figure (d); (j) Partial enlarged view of figure (e))

Table 2 Quantitative analysis of reconstruction results of different algorithms on Mid3 (SSIM)

算法	×2			×3			×4
算法	Art	Books	Moebius	Art	Books	Moebius	Art	Books	Moebius
Bicubic	0.989 8	0.996 1	0.997 3	0.976 8	0.993 4	0.993 9	0.967 4	0.990 7	0.989 2
SRCNN	0.989 8	0.996 1	0.997 6	0.981 0	0.993 9	0.993 6	0.972 7	0.991 1	0.989 4
ESPCN	0.989 9	0.996 1	0.997 7	0.975 9	0.994 0	0.994 3	0.974 0	0.991 2	0.989 7
Ours	0.999 8	0.999 9	0.999 9	0.999 5	0.999 7	0.999 6	0.997 7	0.999 5	0.999 3

References 26

[1]	HORNÁCEK M, RHEMANN C, GELAUTZ M, et al. Depth super resolution by rigid body self-similarity in 3D[C]// 2013 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2013: 1123-1130.
[2]	LEI J J, LI L L, YUE H J, et al. Depth map super-resolution considering view synthesis quality[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2017, 26(4): 1732-1745. DOI URL
[3]	XIE J, FERIS R S, SUN M T. Edge-guided single depth image super resolution[C]// 2014 IEEE International Conference on Image Processing. New York: IEEE Press, 2016: 3773-3777.
[4]	LIU W, CHEN X G, YANG J, et al. Robust color guided depth map restoration[J]. IEEE Transactions on Image Processing, 2017, 26(1): 315-327. PMID
[5]	李滔, 董秀成, 张晓华. 深度图像超分辨率重建技术综述[J]. 西华大学学报: 自然科学版, 2020, 39(4): 45-53.
	LI T, DONG X C, ZHANG X H. A survey of super-resolution reconstruction technologies for depth image[J]. Journal of Xihua University: Natural Science Edition, 2020, 39(4): 45-53. (in Chinese)
[6]	MAC AODHA O, CAMPBELL N D F, NAIR A, et al. Patch based synthesis for single depth image super-resolution[C]// European Conference on Computer Vision. Heidelberg: Springer, 2012: 71-84.
[7]	DONG C, LOY C C, HE K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. DOI PMID
[8]	LI Y J, HUANG J B, AHUJA N, et al. Deep Joint Image Filtering[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2016: 154-169.
[9]	HUI T W, LOY C C, TANG X O. Depth Map Super-Resolution by Deep Multi-Scale Guidance[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2016: 353-369.
[10]	ZHAN Y L, CHI J, YE Y N, et al. Super resolution image reconstruction based on image similarity and feature combination[J]. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(6): 1018-1029.
[11]	ZHENG H T, JI M Q, WANG H Q, et al. CrossNet: an end-to-end reference-based super resolution network using cross-scale warping[M]// Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 88-104.
[12]	YANG F Z, YANG H, FU J L, et al. Learning texture transformer network for image super-resolution[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 5790-5799.
[13]	SUN B L, YE X C, LI B P, et al. Learning scene structure guidance via cross-task knowledge transfer for single depth super-resolution[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 7788-7797.
[14]	ZHENG H T, JI M Q, HAN L, et al. Learning scene structure guidance via cross-task knowledge transfer for single depth super-resolution, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition[M]. New York: IEEE Press, 2021: 7788-7797.
[15]	YUE H J, SUN X Y, YANG J Y, et al. Landmark image super-resolution by retrieving web images[J]. IEEE Transactions on Image Processing, 2013, 22(12): 4865-4878. DOI PMID
[16]	ZHU Y, ZHANG Y N, YUILLE A L. Single image super-resolution using deformable patches[C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2014: 2917-2924.
[17]	XIA B, TIAN Y P, HANG Y C, et al. Coarse-to-fine embedded PatchMatch and multi-scale dynamic aggregation for reference-based super-resolution[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(3): 2768-2776. DOI URL
[18]	ZHANG Z F, WANG Z W, LIN Z, et al. Image super-resolution by neural texture transfer[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 7974-7983.
[19]	GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of Wasserstein GANs[EB/OL]. [2022-12-06]. https://arxiv.org/abs/1704.00028.
[20]	JOHNSON J, ALAHI A, LI F F. Perceptual Losses for Real-Time Style Transfer and Super-Resolution[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2016: 694-711.
[21]	LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 105-114.
[22]	SAJJADI M S M, SCHÖLKOPF B, HIRSCH M. EnhanceNet: single image super-resolution through automated texture synthesis[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 4501-4510.
[23]	ZHANG Z F, WANG Z W, LIN Z, et al. Image super-resolution by neural texture transfer[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 7974-7983.
[24]	范佩佩, 董秀成, 李滔, 等. 基于非局部均值约束的深度图像超分辨率重建[J]. 计算机辅助设计与图形学学报, 2020, 32(10): 1671-1678.
	FAN P P, DONG X C, LI T, et al. Super-resolution reconstruction of depth map based on non-local means constraint[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(10): 1671-1678. (in Chinese)
[25]	LOWE D G. Object recognition from local scale-invariant features[C]// The 7th IEEE International Conference on Computer Vision. New York: IEEE Press, 2002: 1150-1157.
[26]	SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 1874-1883.