A sketch-guided facial image completion network via selective recurrent inference

doi:10.11996/JG.j.2095-302X.2023010067

Abstract

Abstract:

Image inpainting plays a key role in applications such as inpainting old photos and removing face mosaics. There are many problems in the existing deep learning-based inpainting models, such as interference information erroneously affecting the encoder and decoder in generating the result, which weakens inpainting quality, and the probabilistic diversity leading to deviation from users' expectations. To address the problems, a sketch-guided facial image completion network was proposed via selective recurrent inference. A selective recurrent inferential strategy was designed at first. A selection mechanism was introduced to solve the influence of erroneous interference on the inference of the encoder and decoder. Then a sketch-based structural information correction module was added to the skip connection between the encoder and decoder, thereby limiting the deviation of the repair results from users' expected structure. Experimental results on the CelebA-HQ dataset show that the proposed method could outperform other classical network models in terms of evaluation indicators and guidance for generating user-expected content. The experimental results on the manually drawn sketches show that user-specified content could be generated by simple hand-drawn methods, exhibiting certain significance in practical application.

Key words: human face, image inpainting, deep learning, sketch image

CLC Number:

TP391

SHAO Ying-jie, YIN Hui, XIE Ying, HUANG Hua. A sketch-guided facial image completion network via selective recurrent inference[J]. Journal of Graphics, 2023, 44(1): 67-76.

Figures/Tables 11

Fig. 1 Results of Shift-Net[10] and GMCNN[11] models ((a) Ground truth; (b) Input; (c) Shift-Net[10]; (d) GMCNN[11])

Fig. 2 The overall structure of SG-FICNet network

Fig. 3 SG-SIC module ((a) Weight coding layer of structure; (b) Bias coding layer of structure)

Table 1 Experimental results with different values of λperc

评价指标	λ_perc
评价指标	0.05	0.10	0.15	0.20
PSNR	31.474 5	31.739 9	31.553 4	31.320 7

Table 2 Comparing with different mask ratios on the CelebA-HQ dataset

评价指标	模型	掩码比例
评价指标	模型	10%~20%	21%~30%	31%~40%	41%~50%
PSNR	RFR^[7]	33.177 1	29.839 6	27.537 7	25.683 5
	Gated Conv^[14]	32.624 1	29.623 7	27.602 4	25.953 2
	Sc-fegan^[12]	34.174 2	30.804 2	28.359 0	26.268 2
	SG-FICNet (本文)	34.811 3	31.739 9	29.668 5	28.033 3
SSIM	RFR^[7]	0.959 2	0.925 7	0.890 1	0.851 2
	Gated Conv^[14]	0.952 7	0.919 1	0.885 5	0.849 8
	Sc-fegan^[12]	0.962 0	0.932 4	0.900 0	0.863 6
	SG-FICNet (本文)	0.965 1	0.937 4	0.908 8	0.878 2
FID	RFR^[7]	1.132 8	2.116 4	3.331 0	4.788 7
	Gated Conv^[14]	3.092 6	6.009 8	9.482 2	13.271 7
	Sc-fegan^[12]	1.216 2	2.185 6	3.480 6	5.144 3
	SG-FICNet (本文)	0.880 8	1.569 0	2.407 2	3.358 5

Table 3 Repair contrast on square center mask

模型	PSNR	SSIM
RFR^[7]	26.453 2	0.900 6
MS-CAHRBN^[8]	26.755 2	0.895 1
SG-FICNet(无SG-SIC)	26.535 3	0.902 2
SG-FICNet(本文)	29.327 9	0.930 0

Fig. 4 Comparison of facial image inpainting results of sketches generated by different models in HED ((a) Ground truth; (b) Input; (c) Sketch; (d) RFR[7]; (e) Gated Conv[14]; (f) Sc-fegan[12]; (g) SG-FICNet)

Fig. 5 Comparison of facial image inpainting results of different models in artificial hand-painted sketches ((a) Ground truth; (b) Input; (c) Sketch; (d) Gated Conv[14]; (e) Sc-fegan[12]; (f) SG-FICNet)

Table 4 Ablation experiments of SRI and SG-SIC on CelebA-HQ dataset

模型	PSNR	SSIM	FID
基础网络	29.839 6	0.925 7	2.116 4
基础网络+SRI	29.941 9	0.926 3	2.013 3
基础网络+SG-SIC	31.683 6	0.936 7	1.676 0

Fig. 6 Different threshold sketches and repair results ((a) Ground truth; (b) Input; (c) Sketch threshold setting 0.1; (d) Result of sketch threshold setting 0.1; (e) Sketch threshold setting 0.8; (f) Result of sketch threshold setting 0.8)

Fig. 7 Quantitative evaluation results of sketch ablation experiments with different thresholds ((a) PSNR; (b) SSIM)

References 28

[1]	林毓秀, 刘慧, 刘绍玲, 等. 三维CT层间图像超分辨重建与修复[J]. 计算机辅助设计与图形学学报, 2020, 32(6): 919-929.
	LIN Y X, LIU H, LIU S L, et al. Super-resolution reconstruction and inpainting of inter-layer image in 3D CT[J]. Journal of Computer-Aided Design ＆ Computer Graphics, 2020, 32(6): 919-929 (in Chinese).
[2]	杨筱平, 王书文. 基于优先权改进算法的敦煌壁画复杂破损区域修复[J]. 计算机辅助设计与图形学学报, 2011, 23(2): 284-289.
	YANG X P, WANG S W. Dunhuang mural inpainting in intricate disrepaired region based on improvement of priority algorithm[J]. Journal of Computer-Aided Design ＆ Computer Graphics, 2011, 23(2): 284-289 (in Chinese).
[3]	强振平, 何丽波, 陈旭, 等. 深度学习图像修复方法综述[J]. 中国图象图形学报, 2019, 24(3): 447-463.
	QIANG Z P, HE L B, CHEN X, et al. Survey on deep learning image inpainting methods[J]. Journal of Image and Graphics, 2019, 24(3): 447-463 (in Chinese).
[4]	BERTALMIO M, SAPIRO G, CASELLES V, et al. Image inpainting[C]//The 27th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 2000: 417-424.
[5]	CHAN T, SHEN J. Mathematical models for local deterministic inpaintings[R]. Los Angeles: University of California. Department of Mathematics, 2000.
[6]	PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 2536-2544.
[7]	LI J, WANG N, ZHANG L, et al. Recurrent feature reasoning for image inpainting[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 7760-7768.
[8]	杨昊, 余映. 利用通道注意力与分层残差网络的图像修复[J]. 计算机辅助设计与图形学学报, 2021, 33(5): 671-681.
	YANG H, YU Y. Image inpainting using channel attention and hierarchical residual networks[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(5): 671-681 (in Chinese).
[9]	RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Heidelberg:Springer, 2015: 234-241.
[10]	YAN Z, LI X, LI M, et al. Shift-net: image inpainting via deep feature rearrangement[C]//The European Conference on Computer Vision. Cham: Springer International Publishin, 2018: 3-19.
[11]	WANG Y, TAO X, QI X J, et al. Image inpainting via generative multi-column convolutional neural networks[C]// The Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2018: 329-338.
[12]	JO Y, PARK J. Sc-fegan: face editing generative adversarial network with user's sketch and color[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 1745-1753.
[13]	PORTENIER T, HU Q, SZABO A, et al. Faceshop: deep sketch-based face image editing[EB/OL]. [2021-12-09]. https://arxiv.org/pdf/1804.08972.pdf.
[14]	YU J, LIN Z, YANG J, et al. Free-form image inpainting with gated convolution[C]//2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 4471-4480.
[15]	IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14.
[16]	YU J, LIN Z, YANG J, et al. Generative image inpainting with contextual attention[C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 5505-5514.
[17]	LIU G, FA R D, SHIH K, et al. Image inpainting for irregular holes using partial convolutions[C]// The European Conference on Computer Vision. Cham: Springer International Publishin, 2018: 85-100.
[18]	LIU H, WAN Z, HUANG W, et al. Pd-Gan: Probabilistic diverse Gan for image inpainting[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 9371-9381.
[19]	温静, 丁友东, 于冰. 基于上下文门卷积的盲图像修复[J]. 图学学报, 2022, 43(1): 70-78.
	WEN J, DING Y D, YU B. Blind image inpainting based on context gated convolution[J]. Journal of Graphics, 2022, 43(1): 70-78 (in Chinese).
[20]	XIONG W, YU J, LIN Z, et al. Foreground-aware image inpainting[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 5840-5848.
[21]	LI J, HE F, ZHANG L, et al. Progressive reconstruction of visual structure for image inpainting[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 5962-5971.
[22]	GUO Z, CHEN Z, YU T, et al. Progressive image inpainting with full-resolution residual network[C]// The 27th ACM International Conference on Multimedia. New York: ACM, 2019: 2496-2504.
[23]	OH S W, LEE S, LEE J Y, et al. Onion-peel networks for deep video completion[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 4403-4412.
[24]	NAZERI K, NG E, JOSEPH T, et al. Edgeconnect: Generative image inpainting with adversarial edge learning[EB/OL]. [2021-12-09].https://arxiv.org/pdf/1901.00212v3.pdf.
[25]	XIE S, TU Z. Holistically-nested edge detection[C]//2015 IEEE International Conference on Computer Vision. New York: IEEE Press, 2015: 1395-1403.
[26]	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2021-12-09].https://arxiv.org/pdf/1409.1556.pdf.
[27]	KARRAS T, AILA T, LAINE S, et al. Progressive growing of gans for improved quality, stability, and variation[EB/OL]. [2021-12-09].https://arxiv.org/pdf/1710.10196.pdf%C2%A0.
[28]	DIEDERIK P K, ADAM J. A method for stochastic optimization[EB/OL]. [2021-12-09]. https://arxiv.org/pdf/1412.6980.pdf.