Review of image super-resolutionbased on deep learning

doi:10.11996/JG.j.2095-302X.2023010001

Abstract

Abstract:

Super-resolution (SR) is an essential technology in digital image processing that reconstructs and produces a matching high-resolution (HR) image based on the low-resolution (LR) image obtained by an observer, thereby improving the resolution of modern digital images. This technology is of significant research and practical value in fields such as digital image compression and transmission, medical imaging, remote sensing imaging, and video perception and monitoring. With the rapid growth of deep learning, novel solutions for SR challenges can be obtained by combining the latest deep learning algorithms. First, discussions were made on the background, development, and technological value of applying deep learning to SR. Second, a brief overview was provided concerning the fundamental methodology, categorization, and advantages and disadvantages of the classic SR methods. Deep learning-based SR methods were categorized and introduced based on distinct implementation strategies and network types. The application of convolutional neural networks (CNN), residual networks (ResNet), and generative adversarial networks (GAN) in SR was investigated and contrasted. The major evaluation indices and solution methodologies were then presented, and the performance of several SR methods on typical data sets was compared. Finally, the deep learning-based SR method was summarized, and the future development trend was forecasted.

Key words: super-resolution, deep learning, evaluation index, degradation model, datasets

CLC Number:

TP391

LI Hong-an, ZHENG Qiao-xue, TAO Ruo-lin, ZHANG Min, LI Zhan-li, KANG Bao-sheng. Review of image super-resolutionbased on deep learning[J]. Journal of Graphics, 2023, 44(1): 1-15.

Figures/Tables 17

References 58

[1]	MAHAPATRA D, BOZORGTABAR B, GARNAVI R. Image super-resolution using progressive generative adversarial networks for medical image analysis[J]. Computerized Medical Imaging and Graphics, 2019, 71: 30-39. DOI PMID
[2]	MORIN R, BASARAB A, KOUAMÉ D. Alternating direction method of multipliers framework for super-resolution in ultrasound imaging[C]//2012 9th IEEE International Symposium on Biomedical Imaging. New York: IEEE Press, 2012: 1595-1598.
[3]	左艳, 黄钢, 聂生东. 深度学习在医学影像智能处理中的应用与挑战[J]. 中国图象图形学报, 2021, 26(2): 305-315.
	ZUO Y, HUANG G, NIE S D. Application and challenges of deep learning in the intelligent processing of medical images[J]. Journal of Image and Graphics, 2021, 26(2): 305-315 (in Chinese).
[4]	MARIO H J, RUBEN F B, PAOLETTI M E, et al. A new deep generative network for unsupervised remote sensing single-image super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(11): 6792-6810. DOI URL
[5]	HA V K, REN J C, XU X Y, et al. Deep learning based single image super-resolution: a survey[J]. International Journal of Automation and Computing, 2019, 16(4): 413-426. DOI
[6]	LI J C, PEI Z H, ZENG T Y. From beginner to master: a survey for deep learning-based single-image super-resolution[EB/OL]. (2021-01-29) [2022-02-09]. https://arxiv.org/abs/2109.14335.
[7]	HARRIS J L. Diffraction and resolving power[J]. JOSA, 1964, 54(7): 931-936. DOI URL
[8]	TSAI R, HUANG T S. Multiframe image restoration and registration[J]. Advances in Computer Vision and Image Processing, 1984, 1(2): 317-339.
[9]	FARSIU S, ROBINSON M D, ELAD M, et al. Fast and robust multiframe super resolution[J]. IEEE Transactions on Image Processing, 2004, 13(10): 1327-1344. PMID
[10]	PARK J S, LEE S W. An example-based face hallucination method for single-frame, low-resolution facial images[J]. IEEE Transactions on Image Processing, 2008, 17(10): 1806-1816. DOI URL
[11]	YANG W M, ZHANG X C, TIAN Y P, et al. Deep learning for single image super-resolution: a brief review[J]. IEEE Transactions on Multimedia, 2019, 21(12): 3106-3121. DOI URL
[12]	DONG C, LOY C C, HE K M, et al. Learning a deep convolutional network for image super-resolution[C]// European Conference on Computer Vision - ECCV 2014. Cham: Springer International Publishing, 2014: 184-199.
[13]	KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 1646-1654.
[14]	SHI W Z, CABALLERO J, HUSZÁR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 1874-1883.
[15]	LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2017: 1132-1140.
[16]	LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 105-114.
[17]	WANG Z X, GAO G W, LI J C, et al. Lightweight image super-resolution with multi-scale feature interaction network[C]// 2021 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2021: 1-6.
[18]	VAN OUWERKERK J D. Image super-resolution survey[J]. Image and Vision Computing, 2006, 24(10): 1039-1052. DOI URL
[19]	YANG C Y, MA C, YANG M H. Single-image super-resolution: a benchmark[M]//Computer Vision - ECCV 2014. Cham: Springer International Publishing, 2014: 372-386.
[20]	LI X, ORCHARD M T. New edge-directed interpolation[J]. IEEE Transactions on Image Processing, 2001, 10(10): 1521-1527. DOI PMID
[21]	CHEN M J, HUANG C H, LEE W L. A fast edge-oriented algorithm for image interpolation[J]. Image and Vision Computing, 2005, 23(9): 791-798. DOI URL
[22]	PRASHANTH H S, SHASHIDHARA H L, BALASUBRAMANYA MURTHY K N. Image scaling comparison using universal image quality index[C]//2009 International Conference on Advances in Computing, Control, and Telecommunication Technologies. New York: IEEE Press, 2009: 859-863.
[23]	GRIBBON K T, BAILEY D G. A novel approach to real-time bilinear interpolation[C]//Proceedings of DELTA 2004. Second IEEE International Workshop on Electronic Design, Test and Applications. New York: IEEE Press, 2004: 126-131.
[24]	KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1981, 29(6): 1153-1160. DOI URL
[25]	谢海平, 谢凯利, 杨海涛. 图像超分辨率方法研究进展[J]. 计算机工程与应用, 2020, 56(19): 34-41. DOI
	XIE H P, XIE K L, YANG H T. Research progress of image super-resolution methods[J]. Computer Engineering and Applications, 2020, 56(19): 34-41 (in Chinese). DOI
[26]	唐艳秋, 潘泓, 朱亚平, 等. 图像超分辨率重建研究综述[J]. 电子学报, 2020, 48(7): 1407-1420. DOI
	TANG Y Q, PAN H, ZHU Y P, et al. A survey of image super-resolution reconstruction[J]. Acta Electronica Sinica, 2020, 48(7): 1407-1420 (in Chinese).
[27]	DAI S Y, HAN M, WU Y, et al. Bilateral back-projection for single image super resolution[C]// 2007 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2007: 1039-1042.
[28]	ZHANG H C, ZHANG Y N, LI H S, et al. Generative Bayesian image super resolution with natural image prior[J]. IEEE Transactions on Image Processing, 2012, 21(9): 4054-4067. DOI PMID
[29]	CHANG H, YEUNG D Y, XIONG Y M. SRthrough neighbor embedding[EB/OL]. [2022-03-05]. https://ieeexplore.ieee.org/abstract/document/1315043.
[30]	YANG J C, WRIGHT J, HUANG T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873. DOI PMID
[31]	满开亮, 汪友生, 刘继荣. 基于稠密残差网络的图像超分辨率重建算法[J]. 图学学报, 2021, 42(4): 556-562.
	MAN K L, WANG Y S, LIU J R. Image super-resolution reconstruction algorithm based on dense residual network[J]. Journal of Graphics, 2021, 42(4): 556-562 (in Chinese).
[32]	ZHAO T, WU X Q. Pyramid feature attention network for saliency detection[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 3080-3089.
[33]	ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[M]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 294-310.
[34]	ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 2472-2481.
[35]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[M]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 3-19.
[36]	NIU B, WEN W L, REN W Q, et al. Single image super-resolution via a holistic attention network[M]//Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 191-207.
[37]	ROY S K, MANNA S, SONG T C, et al. Attention-based adaptive spectral-spatial kernel ResNet for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(9): 7831-7843. DOI URL
[38]	ZHANG Q, CUI Z P, NIU X G, et al. Image segmentation with pyramid dilated convolution based on ResNet and U-net[M]// Neural Information Processing. Cham: Springer International Publishing, 2017: 364-372.
[39]	LI J C, FANG F M, MEI K F, et al. Multi-scale residual network for image super-resolution[M]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 527-542.
[40]	QIN J H, HUANG Y J, WEN W S. Multi-scale feature fusion residual network for single image super-resolution[J]. Neurocomputing, 2020, 379: 334-342. DOI URL
[41]	BULAT A, YANG J, TZIMIROPOULOS G. To learn image super-resolution, use a GAN to learn how to do image degradation first[M]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 187-202.
[42]	GU J, SHEN Y, ZHOU B. Image processing using multi-code gan prior[EB/OL]. [2022-02-05]. https://ieeexplore.ieee.org/document/9157000.
[43]	YUAN Y, LIU S Y, ZHANG J W, et al. Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2018: 814. 1-814.10
[44]	GU J J, LU H N, ZUO W M, et al. Blind super-resolution with iterative kernel correction[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 1604-1613.
[45]	ZHOU R F, SÜSSTRUNK S. Kernel modeling super-resolution on real low-resolution images[C]//2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 2433-2443.
[46]	PRAJAPATI K, CHUDASAMA V, PATEL H, et al. Unsupervised single image super-resolution network (USISResNet) for real-world data using generative adversarial network[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2020: 1904-1913.
[47]	ZHANG K, LIANG J Y, VAN GOOL L, et al. Designing a practical degradation model for deep blind image super-resolution[C]//2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 4771-4780.
[48]	方玉明, 眭相杰, 鄢杰斌, 等. 无参考图像质量评价研究进展[J]. 中国图象图形学报, 2021, 26(2): 265-286.
	FANG Y M, SUI X J, YAN J B, et al. Progress in no-reference image quality assessment[J]. Journal of Image and Graphics, 2021, 26(2): 265-286 (in Chinese).
[49]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. DOI PMID
[50]	ZHAO H, GALLO O, FROSIO I, et al. Loss functions for image restoration with neural networks[EB/OL]. [2022-02-05]. https://ieeexplore.ieee.org/abstract/document/7797130.
[51]	ZHANG R, ISOLA P, EFROS A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 586-595.
[52]	AGUSTSSON E, TIMOFTE R. NTIRE 2017 challenge on single image super-resolution: dataset and study[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2017: 1122-1131.
[53]	MATSUI Y, ITO K, ARAMAKI Y, et al. Sketch-based manga retrieval using manga109 dataset[J]. Multimedia Tools and Applications, 2017, 76(20): 21811-21838. DOI URL
[54]	CAI J R, ZENG H, YONG H W, et al. Toward real-world single image super-resolution: a new benchmark and a new model[C]//2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 3086-3095.
[55]	PAN Z H, LI B P, XI T, et al. Real image super resolution via heterogeneous model ensemble using GP-NAS[EB/OL]. [2022-02-05]. https://arxiv.org/abs/2009.01371.
[56]	KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 1637-1645.
[57]	MUQEET A, HWANG J, YANG S B, et al. Multi-attention based ultra lightweight image super-resolution[M]//Computer Vision - ECCV 2020 Workshops. Cham: Springer International Publishing, 2020: 103-118.
[58]	ZHOU S C, ZHANG J W, ZUO W M, et al. Cross-scale internal graph neural network for image super- resolution[EB/OL]. [2022-02-05].https://arxiv.org/abs/2006.16673.

算法	Set5		Set14		BSD100
算法	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Bicubic^[23]	33.66	0.929 9	30.24	0.868 8	29.56	0.843 1
SRCNN^[12]	36.66	0.954 2	32.45	0.906 7	31.36	0.887 9
VDSR^[13]	37.53	0.958 7	33.05	0.912 7	31.90	0.896 0
DRCN^[56]	37.63	0.958 8	33.04	0.918 8	31.85	0.894 2
MDSR^[15]	38.11	0.960 2	33.85	0.919 8	32.29	0.900 7
RDN^[33]	38.24	0.961 4	34.01	0.921 2	32.34	0.901 7
MSRN^[39]	38.08	0.960 5	33.74	0.917 0	32.23	0.900 2
MSFFRN^[40]	38.15	0.961 0	33.88	0.919 5	2.29	0.901 0
MAFFSRN^[57]	37.97	0.960 3	33.49	0.917 0	32.14	0.899 4
IGNN^[58]	38.24	0.961 3	34.07	0.921 7	32.41	0.902 5
HAN^[36]	38.27	0.961 4	34.16	0.921 7	32.41	0.902 7

算法	Set5		Set14		BSD100
算法	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Bicubic^[23]	33.66	0.929 9	30.24	0.868 8	29.56	0.843 1
SRCNN^[12]	36.66	0.954 2	32.45	0.906 7	31.36	0.887 9
VDSR^[13]	37.53	0.958 7	33.05	0.912 7	31.90	0.896 0
DRCN^[56]	37.63	0.958 8	33.04	0.918 8	31.85	0.894 2
MDSR^[15]	38.11	0.960 2	33.85	0.919 8	32.29	0.900 7
RDN^[33]	38.24	0.961 4	34.01	0.921 2	32.34	0.901 7
MSRN^[39]	38.08	0.960 5	33.74	0.917 0	32.23	0.900 2
MSFFRN^[40]	38.15	0.961 0	33.88	0.919 5	2.29	0.901 0
MAFFSRN^[57]	37.97	0.960 3	33.49	0.917 0	32.14	0.899 4
IGNN^[58]	38.24	0.961 3	34.07	0.921 7	32.41	0.902 5
HAN^[36]	38.27	0.961 4	34.16	0.921 7	32.41	0.902 7

算法	Set5		Set14		BSD100
算法	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Bicubic^[23]	30.39	0.868 2	27.55	0.774 2	27.21	0.738 5
SRCNN^[12]	32.75	0.909 0	29.30	0.821 5	28.41	0.786 3
VDSR^[13]	33.66	0.921 3	29.78	0.831 8	28.83	0.797 6
DRCN^[56]	33.82	0.922 6	29.76	0.911 8	28.80	0.796 3
MDSR^[15]	38.11	0.960 2	30.44	0.845 2	29.25	0.809 1
RDN^[33]	34.71	0.929 6	30.57	0.846 8	29.26	0.809 3
MSRN^[39]	34.38	0.926 2	30.34	0.839 5	29.08	0.804 1
MSFFRN^[40]	34.65	0.929 2	30.53	0.846 3	29.23	0.808 6
MAFFSRN^[57]	34.32	0.926 9	30.35	0.842 9	29.09	0.805 2
IGNN^[58]	34.72	0.929 8	30.66	0.848 4	29.31	0.810 5
HAN^[36]	34.75	0.929 9	30.67	0.848 3	29.32	0.811 0

算法	Set5		Set14		BSD100
算法	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Bicubic^[23]	30.39	0.868 2	27.55	0.774 2	27.21	0.738 5
SRCNN^[12]	32.75	0.909 0	29.30	0.821 5	28.41	0.786 3
VDSR^[13]	33.66	0.921 3	29.78	0.831 8	28.83	0.797 6
DRCN^[56]	33.82	0.922 6	29.76	0.911 8	28.80	0.796 3
MDSR^[15]	38.11	0.960 2	30.44	0.845 2	29.25	0.809 1
RDN^[33]	34.71	0.929 6	30.57	0.846 8	29.26	0.809 3
MSRN^[39]	34.38	0.926 2	30.34	0.839 5	29.08	0.804 1
MSFFRN^[40]	34.65	0.929 2	30.53	0.846 3	29.23	0.808 6
MAFFSRN^[57]	34.32	0.926 9	30.35	0.842 9	29.09	0.805 2
IGNN^[58]	34.72	0.929 8	30.66	0.848 4	29.31	0.810 5
HAN^[36]	34.75	0.929 9	30.67	0.848 3	29.32	0.811 0

算法	Set5		Set14		BSD100
算法	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Bicubic^[23]	28.42	0.810 4	26.00	0.702 7	25.96	0.667 5
SRCNN^[12]	30.48	0.862 8	27.50	0.751 3	26.90	0.710 1
VDSR^[13]	31.35	0.883 8	28.02	0.767 8	27.29	0.725 2
DRCN^[56]	31.53	0.885 4	28.02	0.767 0	27.23	0.723 3
SRGAN^[16]	32.05	0.891 0	28.53	0.780 4	27.57	0.735 4
MDSR^[15]	32.50	0.897 3	28.72	0.785 7	27.72	0.741 8
RDN^[33]	32.47	0.899 0	28.81	0.787 1	27.72	0.741 9
MSRN^[39]	32.07	0.890 3	28.60	0.775 1	27.52	0.727 3
MSFFRN^[40]	32.44	0.897 8	28.76	0.786 0	27.67	0.740 0
MAFFSRN^[57]	32.18	0.894 8	28.58	0.781 2	27.57	0.736 1
IGNN^[58]	32.57	0.899 8	28.85	0.789 1	27.77	0.743 4
HAN^[36]	32.64	0.900 2	28.90	0.789 0	27.80	0.744 2
MSFIN^[17]	32.28	0.895 7	28.57	0.781 3	27.56	0.735 8