Landscape image generation based on conditional residual generative adversarial network

doi:10.11996/JG.j.2095-302X.2023040710

Abstract

Abstract:

The semantic segmentation map of landscape image encompasses a large number of categorical information such as the sky, white clouds, mountains, rivers, and trees. In view of the challenges presented by the numerous information categories in the semantic segmentation map and the subtle color transformations between different regions, the landscape images generated by current methods are deficient in terms of both clarity and authenticity. Consequently, a method based on conditional residual generation adversarial network (CRGAN) was proposed to generate landscape images with a higher resolution and more realistic content. Firstly, the proposed method involved the upsampling and downsampling structures of the generator network to enhance the feature extraction effect of the generator on the semantic segmentation graph. Secondly, skip connections were utilized between the encoder and decoder to transmit the feature information from the semantic segmentation graph, ensuring the integrity of such information was retained, and not lost in the encoder. Finally, a residual module was added between the encoder and decoder of the network, facilitating better extraction, transmission, and retention of semantic information. In addition, the mean square error (MSE) was employed to enhance the similarity between semantically segmented graphs and generated images. The experimental results demonstrated that compared with pix2pix and cyclegan methods, the FID index of images generated by CRGAN increased by 26.769 and 119.333, respectively. This improvement effectively enhanced the clarity and authenticity of landscape images. The universality and validity of CRGAN were also validated using a common dataset.

Key words: generative adversarial network, landscape image, image generation, deep learning, clarity

CLC Number:

TP391

SHAO Jun-qi, QIAN Wen-hua, XU Qi-hao. Landscape image generation based on conditional residual generative adversarial network[J]. Journal of Graphics, 2023, 44(4): 710-717.

Figures/Tables 10

References 22

[1]	CHEN M H, ZHAO S, LIU H F, et al. Adversarial-learned loss for domain adaptation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(4): 3521-3528. DOI URL
[2]	JIANG X, LAO Q C, MATWIN S, et al. Implicit class-conditioned domain alignment for unsupervised domain adaptation[C]// The 37th International Conference on Machine Learning. New York: ACM, 2020: 4816-4827.
[3]	ZHANG X L, SONG H H, ZHANG K H, et al. Single image super-resolution with enhanced Laplacian pyramid network via conditional generative adversarial learning[J]. Neurocomputing, 2020, 398: 531-538. DOI URL
[4]	KIM H J, LEE D. Image denoising with conditional generative adversarial networks (CGAN) in low dose chest images[J]. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2020, 954: 161914. DOI URL
[5]	JOSE A, FRANCIS A. Reversible colour density compression of images using cGANs[EB/OL]. (2021-06-19) [2022-07-04]. https://arxiv.org/abs/2106.10542.
[6]	黄凯奇, 赵鑫, 李乔哲, 等. 视觉图灵: 从人机对抗看计算机视觉下一步发展[J]. 图学学报, 2021, 42(3): 339-348.
	HUANG K Q, ZHAO X, LI Q Z, et al. Visual Turing: the next development of computer vision in the view of human- computer gaming[J]. Journal of Graphics, 2021, 42(3): 339-348 (in Chinese).
[7]	林晓, 屈时操, 黄伟, 等. 显著区域保留的图像风格迁移算法[J]. 图学学报, 2021, 42(2): 190-197.
	LIN X, QU S C, HUANG W, et al. Style transfer algorithm for salient region preservation[J]. Journal of Graphics, 2021, 42(2): 190-197 (in Chinese).
[8]	任好盼, 王文明, 危德健, 等. 基于高分辨率网络的人体姿态估计方法[J]. 图学学报, 2021, 42(3): 432-438.
	REN H P, WANG W M, WEI D J, et al. Human pose estimation based on high-resolution net[J]. Journal of Graphics, 2021, 42(3): 432-438 (in Chinese).
[9]	李彬, 王平, 赵思逸. 基于双重注意力机制的图像超分辨重建算法[J]. 图学学报, 2021, 42(2): 206-215.
	LI B, WANG P, ZHAO S Y. Image super-resolution reconstruction based on dual attention mechanism[J]. Journal of Graphics, 2021, 42(2): 206-215 (in Chinese).
[10]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144. DOI URL
[11]	KANG M, PARK J. Contragan: contrastive learning for conditional image generation[J]. Advances in Neural Information Processing Systems, 2020, 33(1): 21357-21369.
[12]	KARRAS T, LAINE S, AILA T M. A style-based generator architecture for generative adversarial networks[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 4396-4405.
[13]	MIRZA M, OSINDERO S. Conditional generative adversarial nets[EB/OL]. (2014-11-04) [2022-07-14]. https://arxiv.org/abs/1411.1784.
[14]	MIYATO T, KOYAMA M. cGANs with projection discriminator[EB/OL]. (2018-08-15) [2022-07-14]. https://arxiv.org/abs/1802.05637.
[15]	SALIMANS T, GOODFELLOW I, ZAREMBA W, et al. Improved techniques for training GANs[C]// The 30th International Conference on Neural Information Processing Systems. Barcelona: MIT Press, 2016: 2234-2242.
[16]	ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5967-5976.
[17]	LI C, WAND M. Combining Markov random fields and convolutional neural networks for image synthesis[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 2479-2486.
[18]	HU S M, LIANG D, YANG G Y, et al. Jittor: a novel deep learning framework with meta-operators and unified graph execution[J]. Science China Information Sciences, 2020, 63(12): 1-21.
[19]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2004, 13(4): 600-612. DOI URL
[20]	TALEBI H, MILANFAR P. NIMA: neural image assessment[J]. IEEE Transactions on Image Processing, 2018, 27(8): 3998-4011. DOI URL
[21]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2242-2251.
[22]	CHEN R, HUANG W, HUANG B, et al. Reusing discriminators for encoding: Towards unsupervised image-to-image translation[C]// The 33rd IEEE/CVF conference on computer vision and pattern recognition. Washington: IEEE Computer Society Press, 2020: 8168-8177.

网络	SSIM	NIMA	FID
CGAN	0.7026	5.154±(1.783)	43.813
CGAN+MSE	0.8152	5.652±(1.629)	41.616

网络	SSIM	NIMA	FID
CGAN	0.7026	5.154±(1.783)	43.813
CGAN+MSE	0.8152	5.652±(1.629)	41.616

模块组成结构			评价指标
Front	Backend	Residual	SSIM	FID
3	3	9	0.6919	55.306
3	3	12	0.7263	50.838
4	4	9	0.7441	43.810
4	4	12	0.8152	41.616
5	5	9	0.7714	46.761

模块组成结构			评价指标
Front	Backend	Residual	SSIM	FID
3	3	9	0.6919	55.306
3	3	12	0.7263	50.838
4	4	9	0.7441	43.810
4	4	12	0.8152	41.616
5	5	9	0.7714	46.761

网络	Landscape image			CMP facade
网络	SSIM	NIMA	FID	SSIM	NIMA	FID
cyclegan	0.618 9	5.066±(1.713)	160.949	0.169	5.373±(1.522)	122.999
Pix2pix	0.742 7	5.446±(1.765)	68.385	0.135	5.247±(1.589)	252.194
NICE-GAN	-	-	-	0.120	5.476±(1.650)	100.000
CRGAN	0.815 2	5.652±(1.629)	41.616	0.419	5.562±(1.571)	92.819