基于参考的Transformer纹理迁移深度图像超分辨率重建

doi:10.11996/JG.j.2095-302X.2023050861

图学学报 ›› 2023, Vol. 44 ›› Issue (5): 861-867.DOI: 10.11996/JG.j.2095-302X.2023050861

• 图像处理与计算机视觉 • 上一篇下一篇

基于参考的Transformer纹理迁移深度图像超分辨率重建

杨陈成¹(), 董秀成¹^,²(), 侯兵¹, 张党成¹, 向贤明¹, 冯琪茗¹

1.西华大学电气与电子信息学院，四川成都 611730
2.四川大学锦江学院，四川眉山 620860

收稿日期:2023-01-31 接受日期:2023-05-08 出版日期:2023-10-31 发布日期:2023-10-31
通讯作者: 董秀成(1963-)，男，教授，硕士。主要研究方向为智能信息处理、计算机视觉等。E-mail：dxc136@163.com
作者简介:杨陈成(1998-)，女，硕士研究生。主要研究方向为图像处理、深度学习。E-mail：yangchencheng2017@163.com
基金资助:
国家自然科学基金项目(11872069);四川省中央引导地方科技发展专项(2021ZYD0034);四威高科-西华大学产学研联合实验室项目(2016-YF04-00044-JH)

Reference based transformer texture migrates depth images super resolution reconstruction

YANG Chen-cheng¹(), DONG Xiu-cheng¹^,²(), HOU Bing¹, ZHANG Dang-cheng¹, XIANG Xian-ming¹, FENG Qi-ming¹

1. School of Electrical Engineering and Electronic Information, Xihua University, Chengdu Sichuan 611730, China
2. Jinjiang College, Sichuan University, Meishan Sichuan 620860, China

Received:2023-01-31 Accepted:2023-05-08 Online:2023-10-31 Published:2023-10-31
Contact: DONG Xiu-cheng (1963-), professor, master. His main research interests cover intelligent information processing, computer vision, etc. E-mail：dxc136@163.com
About author:YANG Chen-cheng (1998-), master student. Her main research interests cover image processing and deep learning. E-mail：yangchencheng2017@163.com
Supported by:
National Natural Science Foundation of China(11872069);Central Government Funds of Guiding Local Scientific and Technological Development for Sichuan Province(2021ZYD0034);Siwei Hi-tech-Xihua University Industry-University-Research Joint Laboratory(2016-YF04-00044-JH)

摘要/Abstract

摘要：

深度图像包含场景深度信息，对颜色和光照的变化具有较强的鲁棒性，使得深度图像在立体视觉等领域广泛应用。由于深度传感器性能的局限性以及深度图像成像环境相对复杂，很难直接获取高质量、高分辨率的深度图像。针对重建出现的边缘细节特征不清晰问题，提出一种基于参考的Transformer纹理迁移深度图像超分辨率重建方法。对预处理后的低分辨率深度图像(LR_D)以及参考图像(Ref)特征块，利用归一化内积进行相似度计算，融合Transformer计算相似位置置信度，并结合注意力机制进行纹理迁移，最后与低分辨率深度图像特征结合，提高图像细节清晰度，进一步精确重建结果。实验结果表明，相较于其他方法，该方法结构相似性(SSIM)值更高，主观视觉效果和客观评价指标均得到了明显的改善，重建效果良好。

关键词: 深度学习, 超分辨率重建, 深度图像, Transformer, 注意力机制

Abstract:

Depth images contain scene depth information and exhibit strong robustness to variations in color and lighting, making them widely used in fields such as stereo vision. However, due to the limitations in depth sensor performance and the complexity of imaging environments, it is challenging to directly obtain high-quality, high-resolution depth images. To address the problem of unclear edge details in reconstructed depth images, a reference-based Transformer texture transfer method for deep image super-resolution reconstruction was proposed. For the preprocessed low-resolution depth images (LR_D) and reference images (Ref) feature blocks, similarity calculation was performed using normalized inner product. The method integrated Transformer to calculate the confidence of similarity positions, and combined it with an attention mechanism for texture transfer. Finally, the method combined the features of the low-resolution depth images to improve image detail clarity and further accurately reconstruct the results. The experimental results demonstrated that compared to other methods, the proposed method could achieve higher structural similarity (SSIM) values, and that both subjective visual effects and objective evaluation indicators have been significantly improved, indicating the excellence of the reconstruction performance.

Key words: deep learning, super-resolution reconstruction, depth image, Transformer, attention mechanism

中图分类号:

TP391

杨陈成, 董秀成, 侯兵, 张党成, 向贤明, 冯琪茗. 基于参考的Transformer纹理迁移深度图像超分辨率重建[J]. 图学学报, 2023, 44(5): 861-867.

YANG Chen-cheng, DONG Xiu-cheng, HOU Bing, ZHANG Dang-cheng, XIANG Xian-ming, FENG Qi-ming. Reference based transformer texture migrates depth images super resolution reconstruction[J]. Journal of Graphics, 2023, 44(5): 861-867.

图/表 8

图1 基于参考的Transformer纹理迁移网络结构框图

Fig. 1 Block diagram of texture migration network in Transformer based on reference

图2 LR_D与Ref相似度估计过程图

Fig. 2 LR_D and Ref similarity estimation process diagram

图3 纹理特征迁移过程图

Fig. 3 Texture feature migration process diagram

表1 实验软硬件配置

Table 1 Experimental software and hardware on figuration

名称	实验配置
操作系统	ubantu 20.04
编程语言	Python 3.7.9
深度学习框架	PyTorch 1.8.0
CPU	Intel Xeon Gold 6226
GPU	NVIDIA RTX3090
Cuda	Cuda 11.1
平台	Pycharm-2021.2.1

图4 数据集图像示例((a) HR_D图像；(b) Ref图像)

Fig. 4 Dataset image example ((a) HR_D image; (b) Ref image)

图5 上采样4倍时不同算法在Mid3上的重建效果对比((a)高分辨率深度图像；(b) Bicubic插值结果；(c) SRCNN算法结果；(d) ESPCN算法结果；(e)本文算法结果；(f)图(a)的局部放大图；(g)图(b)的局部放大图；(h)图(c)的局部放大图；(i)图(d)的局部放大图；(j)图(e)的局部放大图)

Fig. 5 Comparison of reconstruction effects of different algorithms on Mid3 when the up-sampling factor is 4 ((a) High resolution depth images; (b) Bicubic interpolation results; (c) SRCNN algorithm results; (d) ESPCN algorithm results; (e) Algorithm results in this article; (f) Partial enlarged view of figure (a); (g) Partial enlarged view of figure (b); (h) Partial enlarged view of figure (c); (i) Partial enlarged view of figure (d); (j) Partial enlarged view of figure (e))

图6 上采样2倍时不同算法在Mid3上的重建效果对比((a)高分辨率深度图像；(b) Bicubic插值结果；(c) SRCNN算法结果；(d) ESPCN算法结果；(e)本文算法结果；(f)图(a)的局部放大图；(g)图(b)的局部放大图；(h)图(c)的局部放大图；(i)图(d)的局部放大图；(j)图(e)的局部放大图)

Fig. 6 Comparison of reconstruction effects of different algorithms on Mid3 when the up-sampling factor is 2 ((a) High resolution depth images; (b) Bicubic interpolation results; (c) SRCNN algorithm results; (d) ESPCN algorithm results; (e) Algorithm results in this article; (f) Partial enlarged view of figure (a); (g) Partial enlarged view of figure (b); (h) Partial enlarged view of figure (c); (i) Partial enlarged view of figure (d); (j) Partial enlarged view of figure (e))

表2 不同算法在Mid3上重建结果的定量分析(SSIM)

Table 2 Quantitative analysis of reconstruction results of different algorithms on Mid3 (SSIM)

算法	×2			×3			×4
算法	Art	Books	Moebius	Art	Books	Moebius	Art	Books	Moebius
Bicubic	0.989 8	0.996 1	0.997 3	0.976 8	0.993 4	0.993 9	0.967 4	0.990 7	0.989 2
SRCNN	0.989 8	0.996 1	0.997 6	0.981 0	0.993 9	0.993 6	0.972 7	0.991 1	0.989 4
ESPCN	0.989 9	0.996 1	0.997 7	0.975 9	0.994 0	0.994 3	0.974 0	0.991 2	0.989 7
Ours	0.999 8	0.999 9	0.999 9	0.999 5	0.999 7	0.999 6	0.997 7	0.999 5	0.999 3

参考文献 26

[1]	HORNÁCEK M, RHEMANN C, GELAUTZ M, et al. Depth super resolution by rigid body self-similarity in 3D[C]// 2013 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2013: 1123-1130.
[2]	LEI J J, LI L L, YUE H J, et al. Depth map super-resolution considering view synthesis quality[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2017, 26(4): 1732-1745. DOI URL
[3]	XIE J, FERIS R S, SUN M T. Edge-guided single depth image super resolution[C]// 2014 IEEE International Conference on Image Processing. New York: IEEE Press, 2016: 3773-3777.
[4]	LIU W, CHEN X G, YANG J, et al. Robust color guided depth map restoration[J]. IEEE Transactions on Image Processing, 2017, 26(1): 315-327. PMID
[5]	李滔, 董秀成, 张晓华. 深度图像超分辨率重建技术综述[J]. 西华大学学报: 自然科学版, 2020, 39(4): 45-53.
	LI T, DONG X C, ZHANG X H. A survey of super-resolution reconstruction technologies for depth image[J]. Journal of Xihua University: Natural Science Edition, 2020, 39(4): 45-53. (in Chinese)
[6]	MAC AODHA O, CAMPBELL N D F, NAIR A, et al. Patch based synthesis for single depth image super-resolution[C]// European Conference on Computer Vision. Heidelberg: Springer, 2012: 71-84.
[7]	DONG C, LOY C C, HE K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. DOI PMID
[8]	LI Y J, HUANG J B, AHUJA N, et al. Deep Joint Image Filtering[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2016: 154-169.
[9]	HUI T W, LOY C C, TANG X O. Depth Map Super-Resolution by Deep Multi-Scale Guidance[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2016: 353-369.
[10]	ZHAN Y L, CHI J, YE Y N, et al. Super resolution image reconstruction based on image similarity and feature combination[J]. Journal of Computer-Aided Design & Computer Graphics, 2019, 31(6): 1018-1029.
[11]	ZHENG H T, JI M Q, WANG H Q, et al. CrossNet: an end-to-end reference-based super resolution network using cross-scale warping[M]// Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 88-104.
[12]	YANG F Z, YANG H, FU J L, et al. Learning texture transformer network for image super-resolution[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 5790-5799.
[13]	SUN B L, YE X C, LI B P, et al. Learning scene structure guidance via cross-task knowledge transfer for single depth super-resolution[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 7788-7797.
[14]	ZHENG H T, JI M Q, HAN L, et al. Learning scene structure guidance via cross-task knowledge transfer for single depth super-resolution, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition[M]. New York: IEEE Press, 2021: 7788-7797.
[15]	YUE H J, SUN X Y, YANG J Y, et al. Landmark image super-resolution by retrieving web images[J]. IEEE Transactions on Image Processing, 2013, 22(12): 4865-4878. DOI PMID
[16]	ZHU Y, ZHANG Y N, YUILLE A L. Single image super-resolution using deformable patches[C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2014: 2917-2924.
[17]	XIA B, TIAN Y P, HANG Y C, et al. Coarse-to-fine embedded PatchMatch and multi-scale dynamic aggregation for reference-based super-resolution[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(3): 2768-2776. DOI URL
[18]	ZHANG Z F, WANG Z W, LIN Z, et al. Image super-resolution by neural texture transfer[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 7974-7983.
[19]	GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of Wasserstein GANs[EB/OL]. [2022-12-06]. https://arxiv.org/abs/1704.00028.
[20]	JOHNSON J, ALAHI A, LI F F. Perceptual Losses for Real-Time Style Transfer and Super-Resolution[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2016: 694-711.
[21]	LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 105-114.
[22]	SAJJADI M S M, SCHÖLKOPF B, HIRSCH M. EnhanceNet: single image super-resolution through automated texture synthesis[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 4501-4510.
[23]	ZHANG Z F, WANG Z W, LIN Z, et al. Image super-resolution by neural texture transfer[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 7974-7983.
[24]	范佩佩, 董秀成, 李滔, 等. 基于非局部均值约束的深度图像超分辨率重建[J]. 计算机辅助设计与图形学学报, 2020, 32(10): 1671-1678.
	FAN P P, DONG X C, LI T, et al. Super-resolution reconstruction of depth map based on non-local means constraint[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(10): 1671-1678. (in Chinese)
[25]	LOWE D G. Object recognition from local scale-invariant features[C]// The 7th IEEE International Conference on Computer Vision. New York: IEEE Press, 2002: 1150-1157.
[26]	SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 1874-1883.

基于参考的Transformer纹理迁移深度图像超分辨率重建

Reference based transformer texture migrates depth images super resolution reconstruction

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 26

相关文章 15

编辑推荐

Metrics

本文评价

[1]	魏陈浩 , 杨睿 , 刘振丙 , 蓝如师 , 孙希延 , 罗笑南 . 具有双层路由注意力的 YOLOv8 道路场景目标检测方法 [J]. 图学学报, 2023, 44(6): 1104-1111.
[2]	丁建川, 肖金桐, 赵可新, 贾冬青, 崔炳德, 杨鑫 . 基于脉冲神经网络的复杂场景导航避障算法 [J]. 图学学报, 2023, 44(6): 1121-1129.
[3]	周锐闯, 田瑾, 闫丰亭, 朱天晓, 张玉金 . 融合外部注意力和图卷积的点云分类模型 [J]. 图学学报, 2023, 44(6): 1162-1172.
[4]	黄少年, 文沛然, 全琪, 陈荣元 . 基于多支路聚合的帧预测轻量化视频异常检测 [J]. 图学学报, 2023, 44(6): 1173-1182.
[5]	张丽媛, 赵海蓉, 何巍, 唐雄风 . 融合全局-局部注意模块的 Mask R-CNN膝关节囊肿检测方法 [J]. 图学学报, 2023, 44(6): 1183-1190.
[6]	石佳豪, 姚莉. 基于语义引导的视频描述生成 [J]. 图学学报, 2023, 44(6): 1191-1201.
[7]	张驰, 张效娟, 赵洋, 杨帆. 基于调色板的半交互式低照度唐卡图像增强 [J]. 图学学报, 2023, 44(6): 1202-1211.
[8]	王吉, 王森, 蒋智文, 谢志峰, 李梦甜. 基于深度条件扩散模型的零样本文本驱动虚拟人生成方法 [J]. 图学学报, 2023, 44(6): 1218-1226.
[9]	党宏社, 许怀彪, 张选德. 融合结构信息的深度学习立体匹配算法[J]. 图学学报, 2023, 44(5): 899-906.
[10]	翟永杰, 郭聪彬, 王乾铭, 赵宽, 白云山, 张冀. 基于隐含空间知识融合的输电线路多金具检测方法[J]. 图学学报, 2023, 44(5): 918-927.
[11]	杨红菊, 高敏, 张常有, 薄文, 武文佳, 曹付元. 一种面向图像修复的局部优化生成模型[J]. 图学学报, 2023, 44(5): 955-965.
[12]	宋焕生, 文雅, 孙士杰, 宋翔宇, 张朝阳, 李旭. 基于改进教师学生网络的隧道火灾检测[J]. 图学学报, 2023, 44(5): 978-987.
[13]	毕春艳, 刘越. 基于深度学习的视频人体动作识别综述[J]. 图学学报, 2023, 44(4): 625-639.
[14]	李利霞, 王鑫, 王军, 张又元. 基于特征融合与注意力机制的无人机图像小目标检测算法[J]. 图学学报, 2023, 44(4): 658-666.
[15]	郝帅, 赵新生, 马旭, 张旭, 何田, 侯李祥. 基于TR-YOLOv5的输电线路多类缺陷目标检测方法[J]. 图学学报, 2023, 44(4): 667-676.