Content semantics and style features match consistent artistic style transfer

doi:10.11996/JG.j.2095-302X.2023040699

Abstract

Abstract:

The development of computer vision has rendered image style transfer a challenging and valuable subject of research. Nonetheless, existing methods are unable to effectively preserve object contours of content images while migrating many different style features with the same content semantics. In response, an artistic style transfer network, with consistent matching of content semantics and style features, was proposed. First, a two-branch feature processing module was employed to enhance the style and content features and retain the object contours of content images. Subsequently, feature distribution alignment and fusion were achieved within the attentional feature space. Finally, an interpolation module with spatial perception capability was utilized to achieve style consistency of content semantics. The network was trained with 82 783 actual photos and 80 095 artistic portraits for style transfer. Furthermore, 1 000 actual photos and 1 000 artistic portraits were used for testing. The effectiveness of the proposed framework and the added loss function was verified through experiments, which included comparing it with the latest four style transfer methods and conducting ablation experiments, respectively. The experimental results demonstrated that the proposed network could run at an average time of 9.42 ms in 256-pixel image generation and 10.23 ms in 512-pixel image generation, while avoiding distortion of content structure and matching content semantics and style features consistently, with better artistic visual effects.

Key words: convolutional neural network, image style transfer, attention mechanism, style consistency, feature fusion

CLC Number:

TP391

LI Xin, PU Yuan-yuan, ZHAO Zheng-peng, XU Dan, QIAN Wen-hua. Content semantics and style features match consistent artistic style transfer[J]. Journal of Graphics, 2023, 44(4): 699-709.

Figures/Tables 9

Fig. 1 General framework of the network

Fig. 2 CSMCNet

Fig. 3 Experimental effect display ((a) Content; (b) Style)

Fig. 4 Comparison with existing methods ((a) Content; (b) Style; (c) Ours; (d) PAMA; (e) AdaAttN; (f) MANet; (g) SANet)

Fig. 5 Loss ablation experiment ((a) Content; (b) Style; (c) w/o Lc; (d) w/o Lrec; (e) w/o Lr; (f) w/o Lh; (g) All loss)

Fig. 6 Structure ablation experiment ((a) Content; (b) Style; (c) w/o whitening; (d) w/o spatial interpolation; (e) w/o fusion module; (f) Ours)

Table 1 Quantitative comparison

方法	FID↓		CF↑	GE↑	LP↑
方法	t-c	t-s	CF↑	GE↑	LP↑
Ours	300.04	506.83	0.57	0.82	0.52
PAMA	483.99	498.73	0.50	0.86	0.50
AdaAttN	357.16	508.73	0.53	0.85	0.49
MANet	499.84	497.21	0.47	0.80	0.48
SANet	483.95	532.29	0.51	0.84	0.50

Table 2 Average running time of image stylization (ms)

方法	时间
方法	256×256	512×512
Ours	9.42	10.23
PAMA	8.53	9.87
AdaAttN	19.76	22.52
MANet	8.20	8.99
SANet	4.79	6.35

Fig. 7 User research

References 33

[1]	ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2242-2251.
[2]	GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 2414-2423.
[3]	GATYS L A, ECKER A S, BETHGE M, et al. Controlling perceptual factors in neural style transfer[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 3730-3738.
[4]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 770-778.
[5]	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90. DOI URL
[6]	CHANDRAN P, ZOSS G, GOTARDO P, et al. Adaptive convolutions for structure-aware style transfer[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 7972-7981.
[7]	WU H, SUN Z X, YUAN W H. Direction-aware neural style transfer[C]// The 26th ACM International Conference on Multimedia. New York: ACM, 2018: 1163-1171.
[8]	ZHI Y H, WEI H W, NI B B. Structure guided photorealistic style transfer[C]// The 26th ACM International Conference on Multimedia. New York: ACM, 2018: 365-373.
[9]	HUANG X, BELONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 1510-1519.
[10]	JING Y C, LIU X, DING Y K, et al. Dynamic instance normalization for arbitrary style transfer[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(4): 4369-4376. DOI URL
[11]	LI Y J, FANG C, YANG J M, et al. Diversified texture synthesis with feed-forward networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 266-274.
[12]	PARK D Y, LEE K H. Arbitrary style transfer with style-attentional networks[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 5873-5881.
[13]	DENG Y Y, TANG F, DONG W M, et al. Arbitrary style transfer via multi-adaptation network[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 2719-2727.
[14]	LIU S H, LIN T W, HE D L, et al. AdaAttN: revisit attention mechanism in arbitrary neural style transfer[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 6629-6638.
[15]	LUO X, HAN Z, YANG L K, et al. Consistent style transfer[EB/OL]. (2022-01-06) [2022-07-16]. https://arxiv.org/abs/2201.02233.
[16]	LIU X C, CHENG M M, LAI Y K, et al. Depth-aware neural style transfer[C]// Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. New York: ACM, 2017: 1-10.
[17]	WANG X, OXHOLM G, ZHANG D, et al. Multimodal transfer: a hierarchical deep convolutional neural network for fast artistic style transfer[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 7178-7186.
[18]	JING Y C, LIU Y, YANG Y Z, et al. Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 244-260.
[19]	KOTOVENKO D, SANAKOYEU A, LANG S, et al. Content and style disentanglement for artistic style transfer[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 4421-4430.
[20]	CHEN D D, YUAN L, LIAO J, et al. StyleBank: an explicit representation for neural image style transfer[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 2770-2779.
[21]	LI Y J, FANG C, YANG J M, et al. Diversified texture synthesis with feed-forward networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 266-274.
[22]	SHENG L, LIN Z Y, SHAO J, et al. Avatar-net: multi-scale zero-shot style transfer by feature decoration[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 8242-8250.
[23]	GU S Y, CHEN C L, LIAO J, et al. Arbitrary style transfer with deep feature reshuffle[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 8222-8231.
[24]	LIU S G, ZHU T. Structure-guided arbitrary style transfer for artistic image and video[J]. IEEE Transactions on Multimedia, 2022, 24: 1299-1312. DOI URL
[25]	LI Y, FANG C, YANG J, et al. Universal style transfer via feature transforms[EB/OL]. [2022-07-15]. https://dl.acm.org/doi/10.5555/3294771.3294808.
[26]	YAO Y, REN J Q, XIE X S, et al. Attention-aware multi-stroke style transfer[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 1467-1475.
[27]	CHEN H, WANG Z, ZHANG H, et al. Artistic style transfer with internal-external learning and contrastive learning[EB/OL]. [2022-07-10]. https://openreview.net/forum?id=hm0i-cunzGW.
[28]	KOLKIN N, SALAVON J, SHAKHNAROVICH G. Style transfer by relaxed optimal transport and self-similarity[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 10043-10052.
[29]	AFIFI M, BRUBAKER M A, BROWN M S. HistoGAN: controlling colors of GAN-generated and real images via color histograms[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 7937-7946.
[30]	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2014: 740-755.
[31]	PHILLIPS F, MACKINTOSH B. Wiki art gallery, inc.: a case for critical thinking[J]. Issues in Accounting Education, 2011, 26(3): 593-608. DOI URL
[32]	KINGMA D P, BA J. Adam: A method for stochastic optimization[EB/OL]. (2017-01-30) [2022-07-17]. https://arxiv.org/abs/1412.6980.
[33]	WANG Z Z, ZHAO L, CHEN H B, et al. Evaluate and improve the quality of neural style transfer[J]. Computer Vision and Image Understanding, 2021, 207: 103203. DOI URL