基于条件残差生成对抗网络的风景图生成

doi:10.11996/JG.j.2095-302X.2023040710

图学学报 ›› 2023, Vol. 44 ›› Issue (4): 710-717.DOI: 10.11996/JG.j.2095-302X.2023040710

• 图像处理与计算机视觉 • 上一篇下一篇

基于条件残差生成对抗网络的风景图生成

邵俊棋(), 钱文华(), 徐启豪

云南大学信息学院计算机科学与工程系，云南昆明 650504

收稿日期:2022-11-17 接受日期:2023-02-21 出版日期:2023-08-31 发布日期:2023-08-16
通讯作者: 钱文华(1980-)，男，教授，博士。主要研究方向为图像处理和计算机视觉等。E-mail：whqian@ynu.edu.cn
作者简介:
邵俊棋(1997-)，男，硕士研究生。主要研究方向为计算机视觉和图像生成。E-mail：619702616@qq.com
基金资助:
国家自然科学基金项目(62162065);云南省科技厅应用基础研究计划重点项目(2019FA044);云南省中青年学术技术带头人后备人才项目(2019HB121);云南大学研究生科研创新项目(ZC-22222502)

Landscape image generation based on conditional residual generative adversarial network

SHAO Jun-qi(), QIAN Wen-hua(), XU Qi-hao

Department of Computer Science Engineering, School of Information Science and Engineering, Yunnan University, Kunming Yunnan 650504, China

Received:2022-11-17 Accepted:2023-02-21 Online:2023-08-31 Published:2023-08-16
Contact: QIAN Wen-hua (1980-), professor, Ph.D. His main research interests cover graphic image processing and computer vision, etc. E-mail：whqian@ynu.edu.cn
About author:
SHAO Jun-qi (1997-), master student. His main research interests cover computer vision and image generation. E-mail：619702616@qq.com
Supported by:
National Natural Science Foundation of China(62162065);Key Project of Applied Basic Research Plan of Yunnan Provincial Department of Science and Technology(2019FA044);Yunnan Young and Middle-Aged Academic and Technical Leaders Reserve Talents Project(2019HB121);Postgraduate Research and Innovation Foundation of Yunnan University(ZC-22222502)

摘要/Abstract

摘要：

风景图像的语义分割图中包含天空、白云、山川、树木、河流等大量类别信息，针对语义分割图中存在的信息类别过多、不同区域间的色彩变换不明显等问题，现有方法生成的风景图像在清晰度和真实性上效果并不理想。因此提出了一种基于条件残差生成对抗网络(CRGAN)方法，用于生成清晰度更高和内容更真实的风景图像。首先，优化生成器网络的上采样和下采样结构，提升生成器对语义分割图的特征提取效果。其次，在编码器和解码器之间使用跳跃连接传递语义分割图的特征信息，防止特征信息在编码器中传递丢失，保留特征信息的完整性。最后，在网络的编码器和解码器之间添加残差模块，以便更好地提取、传输和保留语义信息。此外，方法中采用均方差(MSE)提升语义分割图和生成图像之间的相似度。实验结果表明，相较于pix2pix和cyclegan方法，CRGAN生成的图像在FID指标中分别增加了26.769和119.333，有效提升了风景图像的清晰度和真实性。同时使用公共数据集验证了CRGAN的泛用性和有效性。

关键词: 生成对抗网络, 风景图像, 图像生成, 深度学习, 清晰度

Abstract:

The semantic segmentation map of landscape image encompasses a large number of categorical information such as the sky, white clouds, mountains, rivers, and trees. In view of the challenges presented by the numerous information categories in the semantic segmentation map and the subtle color transformations between different regions, the landscape images generated by current methods are deficient in terms of both clarity and authenticity. Consequently, a method based on conditional residual generation adversarial network (CRGAN) was proposed to generate landscape images with a higher resolution and more realistic content. Firstly, the proposed method involved the upsampling and downsampling structures of the generator network to enhance the feature extraction effect of the generator on the semantic segmentation graph. Secondly, skip connections were utilized between the encoder and decoder to transmit the feature information from the semantic segmentation graph, ensuring the integrity of such information was retained, and not lost in the encoder. Finally, a residual module was added between the encoder and decoder of the network, facilitating better extraction, transmission, and retention of semantic information. In addition, the mean square error (MSE) was employed to enhance the similarity between semantically segmented graphs and generated images. The experimental results demonstrated that compared with pix2pix and cyclegan methods, the FID index of images generated by CRGAN increased by 26.769 and 119.333, respectively. This improvement effectively enhanced the clarity and authenticity of landscape images. The universality and validity of CRGAN were also validated using a common dataset.

Key words: generative adversarial network, landscape image, image generation, deep learning, clarity

中图分类号:

TP391

邵俊棋, 钱文华, 徐启豪. 基于条件残差生成对抗网络的风景图生成[J]. 图学学报, 2023, 44(4): 710-717.

SHAO Jun-qi, QIAN Wen-hua, XU Qi-hao. Landscape image generation based on conditional residual generative adversarial network[J]. Journal of Graphics, 2023, 44(4): 710-717.

图/表 10

图1 网络模型

Fig. 1 Network model

图2 生成器网络模型

Fig. 2 Generator network model

图3 Front block结构

Fig. 3 Front block structure

图4 CGAN网络模型

Fig. 4 CGAN network model

图5 增加MSE后结果对比

Fig. 5 Comparison of results after adding MSE ((a) Input; (b) CGAN; (c) CGAN+MSE)

表1 增加MSE函数后指标对比

Table 1 Comparison after adding MSE function

网络	SSIM	NIMA	FID
CGAN	0.7026	5.154±(1.783)	43.813
CGAN+MSE	0.8152	5.652±(1.629)	41.616

表2 不同数量的卷积块和残差块对实验结果的影响

Table 2 The effect of different numbers of convolutional blocks and residual blocks on the experimental results

模块组成结构			评价指标
Front	Backend	Residual	SSIM	FID
3	3	9	0.6919	55.306
3	3	12	0.7263	50.838
4	4	9	0.7441	43.810
4	4	12	0.8152	41.616
5	5	9	0.7714	46.761

图6 本文与其他方法的对比

Fig. 6 Comparison with other methods ((a) Input; (b) Pix2pix; (c) Cycegan; (d) Ours)

图7 本文与其他方法的对比

Fig. 7 Comparison with other methods ((a) Input; (b) Pix2pix; (c) Cycegan; (d) NICE-GAN; (e) Ours)

表3 方法指标对比分析

Table 3 Method comparative analysis

网络	Landscape image			CMP facade
网络	SSIM	NIMA	FID	SSIM	NIMA	FID
cyclegan	0.618 9	5.066±(1.713)	160.949	0.169	5.373±(1.522)	122.999
Pix2pix	0.742 7	5.446±(1.765)	68.385	0.135	5.247±(1.589)	252.194
NICE-GAN	-	-	-	0.120	5.476±(1.650)	100.000
CRGAN	0.815 2	5.652±(1.629)	41.616	0.419	5.562±(1.571)	92.819

参考文献 22

[1]	CHEN M H, ZHAO S, LIU H F, et al. Adversarial-learned loss for domain adaptation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(4): 3521-3528. DOI URL
[2]	JIANG X, LAO Q C, MATWIN S, et al. Implicit class-conditioned domain alignment for unsupervised domain adaptation[C]// The 37th International Conference on Machine Learning. New York: ACM, 2020: 4816-4827.
[3]	ZHANG X L, SONG H H, ZHANG K H, et al. Single image super-resolution with enhanced Laplacian pyramid network via conditional generative adversarial learning[J]. Neurocomputing, 2020, 398: 531-538. DOI URL
[4]	KIM H J, LEE D. Image denoising with conditional generative adversarial networks (CGAN) in low dose chest images[J]. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 2020, 954: 161914. DOI URL
[5]	JOSE A, FRANCIS A. Reversible colour density compression of images using cGANs[EB/OL]. (2021-06-19) [2022-07-04]. https://arxiv.org/abs/2106.10542.
[6]	黄凯奇, 赵鑫, 李乔哲, 等. 视觉图灵: 从人机对抗看计算机视觉下一步发展[J]. 图学学报, 2021, 42(3): 339-348.
	HUANG K Q, ZHAO X, LI Q Z, et al. Visual Turing: the next development of computer vision in the view of human- computer gaming[J]. Journal of Graphics, 2021, 42(3): 339-348 (in Chinese).
[7]	林晓, 屈时操, 黄伟, 等. 显著区域保留的图像风格迁移算法[J]. 图学学报, 2021, 42(2): 190-197.
	LIN X, QU S C, HUANG W, et al. Style transfer algorithm for salient region preservation[J]. Journal of Graphics, 2021, 42(2): 190-197 (in Chinese).
[8]	任好盼, 王文明, 危德健, 等. 基于高分辨率网络的人体姿态估计方法[J]. 图学学报, 2021, 42(3): 432-438.
	REN H P, WANG W M, WEI D J, et al. Human pose estimation based on high-resolution net[J]. Journal of Graphics, 2021, 42(3): 432-438 (in Chinese).
[9]	李彬, 王平, 赵思逸. 基于双重注意力机制的图像超分辨重建算法[J]. 图学学报, 2021, 42(2): 206-215.
	LI B, WANG P, ZHAO S Y. Image super-resolution reconstruction based on dual attention mechanism[J]. Journal of Graphics, 2021, 42(2): 206-215 (in Chinese).
[10]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144. DOI URL
[11]	KANG M, PARK J. Contragan: contrastive learning for conditional image generation[J]. Advances in Neural Information Processing Systems, 2020, 33(1): 21357-21369.
[12]	KARRAS T, LAINE S, AILA T M. A style-based generator architecture for generative adversarial networks[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 4396-4405.
[13]	MIRZA M, OSINDERO S. Conditional generative adversarial nets[EB/OL]. (2014-11-04) [2022-07-14]. https://arxiv.org/abs/1411.1784.
[14]	MIYATO T, KOYAMA M. cGANs with projection discriminator[EB/OL]. (2018-08-15) [2022-07-14]. https://arxiv.org/abs/1802.05637.
[15]	SALIMANS T, GOODFELLOW I, ZAREMBA W, et al. Improved techniques for training GANs[C]// The 30th International Conference on Neural Information Processing Systems. Barcelona: MIT Press, 2016: 2234-2242.
[16]	ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5967-5976.
[17]	LI C, WAND M. Combining Markov random fields and convolutional neural networks for image synthesis[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 2479-2486.
[18]	HU S M, LIANG D, YANG G Y, et al. Jittor: a novel deep learning framework with meta-operators and unified graph execution[J]. Science China Information Sciences, 2020, 63(12): 1-21.
[19]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2004, 13(4): 600-612. DOI URL
[20]	TALEBI H, MILANFAR P. NIMA: neural image assessment[J]. IEEE Transactions on Image Processing, 2018, 27(8): 3998-4011. DOI URL
[21]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2242-2251.
[22]	CHEN R, HUANG W, HUANG B, et al. Reusing discriminators for encoding: Towards unsupervised image-to-image translation[C]// The 33rd IEEE/CVF conference on computer vision and pattern recognition. Washington: IEEE Computer Society Press, 2020: 8168-8177.

基于条件残差生成对抗网络的风景图生成

Landscape image generation based on conditional residual generative adversarial network

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 22

相关文章 15

编辑推荐

Metrics

本文评价

[1]	杨陈成 , 董秀成 , 侯兵 , 张党成 , 向贤明 , 冯琪茗 . 基于参考的Transformer纹理迁移深度图像超分辨率重建 [J]. 图学学报, 2023, 44(5): 861-867.
[2]	党宏社 , 许怀彪 , 张选德 . 融合结构信息的深度学习立体匹配算法 [J]. 图学学报, 2023, 44(5): 899-906.
[3]	翟永杰, 郭聪彬, 王乾铭, 赵宽, 白云山, 张冀 . 基于隐含空间知识融合的输电线路多金具检测方法 [J]. 图学学报, 2023, 44(5): 918-927.
[4]	徐祯东, 张天宇, 张世恒, 姚从荣, 王道累. 基于YUV颜色空间GAN网络的图像去雾算法研究 [J]. 图学学报, 2023, 44(5): 927-936.
[5]	杨红菊, 高敏, 张常有, 薄文, 武文佳, 曹付元. 一种面向图像修复的局部优化生成模型 [J]. 图学学报, 2023, 44(5): 955-965.
[6]	毕春艳, 刘越. 基于深度学习的视频人体动作识别综述[J]. 图学学报, 2023, 44(4): 625-639.
[7]	曹义亲, 周一纬, 徐露. 基于E-YOLOX的实时金属表面缺陷检测算法[J]. 图学学报, 2023, 44(4): 677-690.
[8]	余伟群, 刘佳涛, 张亚萍. 融合注意力的拉普拉斯金字塔单目深度估计[J]. 图学学报, 2023, 44(4): 728-738.
[9]	郭印宏, 王立春, 李爽. 基于重复性和特异性约束的图像特征匹配[J]. 图学学报, 2023, 44(4): 739-746.
[10]	毛爱坤, 刘昕明, 陈文壮, 宋绍楼. 改进YOLOv5算法的变电站仪表目标检测方法[J]. 图学学报, 2023, 44(3): 448-455.
[11]	王佳婧, 王晨, 朱媛媛, 王笑梅. 基于民国纸币的图元素匹配检索[J]. 图学学报, 2023, 44(3): 492-501.
[12]	杨柳, 吴晓群. 基于深度学习的三维形状补全研究综述[J]. 图学学报, 2023, 44(2): 201-215.
[13]	曾武, 朱恒亮, 邢树礼, 林江宏, 毛国君. 显著性检测引导的图像数据增强方法[J]. 图学学报, 2023, 44(2): 260-270.
[14]	罗启明, 吴昊, 夏信, 袁国武. 基于Dual Dense U-Net的云南壁画破损区域预测[J]. 图学学报, 2023, 44(2): 304-312.
[15]	李洪安, 郑峭雪, 陶若霖, 张敏, 李占利, 康宝生. 基于深度学习的图像超分辨率研究综述[J]. 图学学报, 2023, 44(1): 1-15.