图学学报 ›› 2023, Vol. 44 ›› Issue (1): 67-76.DOI: 10.11996/JG.j.2095-302X.2023010067
邵英杰1,2(), 尹辉1,2(), 谢颖1,2, 黄华1,2
收稿日期:
2022-06-19
修回日期:
2022-07-20
出版日期:
2023-10-31
发布日期:
2023-02-16
通讯作者:
尹辉
作者简介:
邵英杰(1998-),男,硕士研究生。主要研究方向为计算机视觉、深度学习。E-mail:906612726@qq.com
基金资助:
SHAO Ying-jie1,2(), YIN Hui1,2(), XIE Ying1,2, HUANG Hua1,2
Received:
2022-06-19
Revised:
2022-07-20
Online:
2023-10-31
Published:
2023-02-16
Contact:
YIN Hui
About author:
SHAO Ying-jie (1998-), master student. His main research interests cover computer vision and deep learning. E-mail:906612726@qq.com
Supported by:
摘要:
图像修复在修复老照片、消除人脸马赛克等应用中起到关键作用。针对现有深度学习人脸图像修复方法因受干扰信息影响,存在编解码器修复效果欠佳,修复结果因概率多样性出现偏离用户预期等问题。提出了一种草图引导的选择循环推理式人脸图像修复网络,通过设计一种选择循环推理策略,在循环网络中引入选择机制降低干扰信息对编解码的推理影响,并在编码器和解码器之间的跳跃连接中加入基于草图的结构信息修正模块,从而限制修复结果相对于用户期望的结构偏离。在CelebA-HQ数据集上的实验结果表明,该方法在评价指标和引导生成用户期望内容方面均优于其他经典网络。在人工手绘草图上的实验结果表明,可以通过简单的手绘方式生成用户指定的内容,具有一定的实际应用意义。
中图分类号:
邵英杰, 尹辉, 谢颖, 黄华. 草图引导的选择循环推理式人脸图像修复网络[J]. 图学学报, 2023, 44(1): 67-76.
SHAO Ying-jie, YIN Hui, XIE Ying, HUANG Hua. A sketch-guided facial image completion network via selective recurrent inference[J]. Journal of Graphics, 2023, 44(1): 67-76.
图1 Shift-Net[10]和GMCNN[11]模型的生成结果((a)原图;(b)输入;(c) Shift-Net[10];(d) GMCNN[11])
Fig. 1 Results of Shift-Net[10] and GMCNN[11] models ((a) Ground truth; (b) Input; (c) Shift-Net[10]; (d) GMCNN[11])
评价指标 | λperc | |||
---|---|---|---|---|
0.05 | 0.10 | 0.15 | 0.20 | |
PSNR | 31.474 5 | 31.739 9 | 31.553 4 | 31.320 7 |
表1 不同λperc值的实验结果
Table 1 Experimental results with different values of λperc
评价指标 | λperc | |||
---|---|---|---|---|
0.05 | 0.10 | 0.15 | 0.20 | |
PSNR | 31.474 5 | 31.739 9 | 31.553 4 | 31.320 7 |
评价指标 | 模型 | 掩码比例 | |||
---|---|---|---|---|---|
10%~20% | 21%~30% | 31%~40% | 41%~50% | ||
PSNR | RFR[ | 33.177 1 | 29.839 6 | 27.537 7 | 25.683 5 |
Gated Conv[ | 32.624 1 | 29.623 7 | 27.602 4 | 25.953 2 | |
Sc-fegan[ | 34.174 2 | 30.804 2 | 28.359 0 | 26.268 2 | |
SG-FICNet (本文) | 34.811 3 | 31.739 9 | 29.668 5 | 28.033 3 | |
SSIM | RFR[ | 0.959 2 | 0.925 7 | 0.890 1 | 0.851 2 |
Gated Conv[ | 0.952 7 | 0.919 1 | 0.885 5 | 0.849 8 | |
Sc-fegan[ | 0.962 0 | 0.932 4 | 0.900 0 | 0.863 6 | |
SG-FICNet (本文) | 0.965 1 | 0.937 4 | 0.908 8 | 0.878 2 | |
FID | RFR[ | 1.132 8 | 2.116 4 | 3.331 0 | 4.788 7 |
Gated Conv[ | 3.092 6 | 6.009 8 | 9.482 2 | 13.271 7 | |
Sc-fegan[ | 1.216 2 | 2.185 6 | 3.480 6 | 5.144 3 | |
SG-FICNet (本文) | 0.880 8 | 1.569 0 | 2.407 2 | 3.358 5 |
表2 在CelebA-HQ数据集上按照不同的掩码比例修复对比
Table 2 Comparing with different mask ratios on the CelebA-HQ dataset
评价指标 | 模型 | 掩码比例 | |||
---|---|---|---|---|---|
10%~20% | 21%~30% | 31%~40% | 41%~50% | ||
PSNR | RFR[ | 33.177 1 | 29.839 6 | 27.537 7 | 25.683 5 |
Gated Conv[ | 32.624 1 | 29.623 7 | 27.602 4 | 25.953 2 | |
Sc-fegan[ | 34.174 2 | 30.804 2 | 28.359 0 | 26.268 2 | |
SG-FICNet (本文) | 34.811 3 | 31.739 9 | 29.668 5 | 28.033 3 | |
SSIM | RFR[ | 0.959 2 | 0.925 7 | 0.890 1 | 0.851 2 |
Gated Conv[ | 0.952 7 | 0.919 1 | 0.885 5 | 0.849 8 | |
Sc-fegan[ | 0.962 0 | 0.932 4 | 0.900 0 | 0.863 6 | |
SG-FICNet (本文) | 0.965 1 | 0.937 4 | 0.908 8 | 0.878 2 | |
FID | RFR[ | 1.132 8 | 2.116 4 | 3.331 0 | 4.788 7 |
Gated Conv[ | 3.092 6 | 6.009 8 | 9.482 2 | 13.271 7 | |
Sc-fegan[ | 1.216 2 | 2.185 6 | 3.480 6 | 5.144 3 | |
SG-FICNet (本文) | 0.880 8 | 1.569 0 | 2.407 2 | 3.358 5 |
模型 | PSNR | SSIM |
---|---|---|
RFR[ | 26.453 2 | 0.900 6 |
MS-CAHRBN[ | 26.755 2 | 0.895 1 |
SG-FICNet(无SG-SIC) | 26.535 3 | 0.902 2 |
SG-FICNet(本文) | 29.327 9 | 0.930 0 |
表3 在正方形中心掩码上的修复对比
Table 3 Repair contrast on square center mask
模型 | PSNR | SSIM |
---|---|---|
RFR[ | 26.453 2 | 0.900 6 |
MS-CAHRBN[ | 26.755 2 | 0.895 1 |
SG-FICNet(无SG-SIC) | 26.535 3 | 0.902 2 |
SG-FICNet(本文) | 29.327 9 | 0.930 0 |
图4 不同模型在HED生成草图的人脸图像修复结果对比((a)原图;(b)输入;(c)草图;(d) RFR[7];(e) Gated Conv[14];(f) Sc-fegan[12];(g) SG-FICNet)
Fig. 4 Comparison of facial image inpainting results of sketches generated by different models in HED ((a) Ground truth; (b) Input; (c) Sketch; (d) RFR[7]; (e) Gated Conv[14]; (f) Sc-fegan[12]; (g) SG-FICNet)
图5 不同模型在人工手绘草图的人脸图像修复结果对比((a)原图;(b)输入;(c)草图;(d) Gated Conv[14];(e) Sc-fegan[12];(f) SG-FICNet)
Fig. 5 Comparison of facial image inpainting results of different models in artificial hand-painted sketches ((a) Ground truth; (b) Input; (c) Sketch; (d) Gated Conv[14]; (e) Sc-fegan[12]; (f) SG-FICNet)
模型 | PSNR | SSIM | FID |
---|---|---|---|
基础网络 | 29.839 6 | 0.925 7 | 2.116 4 |
基础网络+SRI | 29.941 9 | 0.926 3 | 2.013 3 |
基础网络+SG-SIC | 31.683 6 | 0.936 7 | 1.676 0 |
表4 SRI和SG-SIC在CelebA-HQ数据集上的消融实验
Table 4 Ablation experiments of SRI and SG-SIC on CelebA-HQ dataset
模型 | PSNR | SSIM | FID |
---|---|---|---|
基础网络 | 29.839 6 | 0.925 7 | 2.116 4 |
基础网络+SRI | 29.941 9 | 0.926 3 | 2.013 3 |
基础网络+SG-SIC | 31.683 6 | 0.936 7 | 1.676 0 |
图6 不同阈值草图及修复结果((a)原图;(b)输入;(c)阈值0.1草图;(d)阈值0.1草图修复图;(e)阈值0.8草图;(f)阈值0.8草图修复图)
Fig. 6 Different threshold sketches and repair results ((a) Ground truth; (b) Input; (c) Sketch threshold setting 0.1; (d) Result of sketch threshold setting 0.1; (e) Sketch threshold setting 0.8; (f) Result of sketch threshold setting 0.8)
[1] | 林毓秀, 刘慧, 刘绍玲, 等. 三维CT层间图像超分辨重建与修复[J]. 计算机辅助设计与图形学学报, 2020, 32(6): 919-929. |
LIN Y X, LIU H, LIU S L, et al. Super-resolution reconstruction and inpainting of inter-layer image in 3D CT[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(6): 919-929 (in Chinese). | |
[2] | 杨筱平, 王书文. 基于优先权改进算法的敦煌壁画复杂破损区域修复[J]. 计算机辅助设计与图形学学报, 2011, 23(2): 284-289. |
YANG X P, WANG S W. Dunhuang mural inpainting in intricate disrepaired region based on improvement of priority algorithm[J]. Journal of Computer-Aided Design & Computer Graphics, 2011, 23(2): 284-289 (in Chinese). | |
[3] | 强振平, 何丽波, 陈旭, 等. 深度学习图像修复方法综述[J]. 中国图象图形学报, 2019, 24(3): 447-463. |
QIANG Z P, HE L B, CHEN X, et al. Survey on deep learning image inpainting methods[J]. Journal of Image and Graphics, 2019, 24(3): 447-463 (in Chinese). | |
[4] | BERTALMIO M, SAPIRO G, CASELLES V, et al. Image inpainting[C]//The 27th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 2000: 417-424. |
[5] | CHAN T, SHEN J. Mathematical models for local deterministic inpaintings[R]. Los Angeles: University of California. Department of Mathematics, 2000. |
[6] | PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 2536-2544. |
[7] | LI J, WANG N, ZHANG L, et al. Recurrent feature reasoning for image inpainting[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 7760-7768. |
[8] | 杨昊, 余映. 利用通道注意力与分层残差网络的图像修复[J]. 计算机辅助设计与图形学学报, 2021, 33(5): 671-681. |
YANG H, YU Y. Image inpainting using channel attention and hierarchical residual networks[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(5): 671-681 (in Chinese). | |
[9] | RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Heidelberg:Springer, 2015: 234-241. |
[10] | YAN Z, LI X, LI M, et al. Shift-net: image inpainting via deep feature rearrangement[C]//The European Conference on Computer Vision. Cham: Springer International Publishin, 2018: 3-19. |
[11] | WANG Y, TAO X, QI X J, et al. Image inpainting via generative multi-column convolutional neural networks[C]// The Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2018: 329-338. |
[12] | JO Y, PARK J. Sc-fegan: face editing generative adversarial network with user's sketch and color[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 1745-1753. |
[13] | PORTENIER T, HU Q, SZABO A, et al. Faceshop: deep sketch-based face image editing[EB/OL]. [2021-12-09]. https://arxiv.org/pdf/1804.08972.pdf. |
[14] | YU J, LIN Z, YANG J, et al. Free-form image inpainting with gated convolution[C]//2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 4471-4480. |
[15] | IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14. |
[16] | YU J, LIN Z, YANG J, et al. Generative image inpainting with contextual attention[C]// 2018 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 5505-5514. |
[17] | LIU G, FA R D, SHIH K, et al. Image inpainting for irregular holes using partial convolutions[C]// The European Conference on Computer Vision. Cham: Springer International Publishin, 2018: 85-100. |
[18] | LIU H, WAN Z, HUANG W, et al. Pd-Gan: Probabilistic diverse Gan for image inpainting[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 9371-9381. |
[19] | 温静, 丁友东, 于冰. 基于上下文门卷积的盲图像修复[J]. 图学学报, 2022, 43(1): 70-78. |
WEN J, DING Y D, YU B. Blind image inpainting based on context gated convolution[J]. Journal of Graphics, 2022, 43(1): 70-78 (in Chinese). | |
[20] | XIONG W, YU J, LIN Z, et al. Foreground-aware image inpainting[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 5840-5848. |
[21] | LI J, HE F, ZHANG L, et al. Progressive reconstruction of visual structure for image inpainting[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 5962-5971. |
[22] | GUO Z, CHEN Z, YU T, et al. Progressive image inpainting with full-resolution residual network[C]// The 27th ACM International Conference on Multimedia. New York: ACM, 2019: 2496-2504. |
[23] | OH S W, LEE S, LEE J Y, et al. Onion-peel networks for deep video completion[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 4403-4412. |
[24] | NAZERI K, NG E, JOSEPH T, et al. Edgeconnect: Generative image inpainting with adversarial edge learning[EB/OL]. [2021-12-09].https://arxiv.org/pdf/1901.00212v3.pdf. |
[25] | XIE S, TU Z. Holistically-nested edge detection[C]//2015 IEEE International Conference on Computer Vision. New York: IEEE Press, 2015: 1395-1403. |
[26] | SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2021-12-09].https://arxiv.org/pdf/1409.1556.pdf. |
[27] | KARRAS T, AILA T, LAINE S, et al. Progressive growing of gans for improved quality, stability, and variation[EB/OL]. [2021-12-09].https://arxiv.org/pdf/1710.10196.pdf%C2%A0. |
[28] | DIEDERIK P K, ADAM J. A method for stochastic optimization[EB/OL]. [2021-12-09]. https://arxiv.org/pdf/1412.6980.pdf. |
[1] | 杨陈成, 董秀成, 侯兵, 张党成, 向贤明, 冯琪茗. 基于参考的Transformer纹理迁移深度图像超分辨率重建[J]. 图学学报, 2023, 44(5): 861-867. |
[2] | 党宏社, 许怀彪, 张选德. 融合结构信息的深度学习立体匹配算法[J]. 图学学报, 2023, 44(5): 899-906. |
[3] | 翟永杰, 郭聪彬, 王乾铭, 赵宽, 白云山, 张冀. 基于隐含空间知识融合的输电线路多金具检测方法[J]. 图学学报, 2023, 44(5): 918-927. |
[4] | 杨红菊, 高敏, 张常有, 薄文, 武文佳, 曹付元. 一种面向图像修复的局部优化生成模型[J]. 图学学报, 2023, 44(5): 955-965. |
[5] | 王可欣, 金映含, 张东亮. 基于深度相机的虚拟眼镜试戴[J]. 图学学报, 2023, 44(5): 988-996. |
[6] | 毕春艳, 刘越. 基于深度学习的视频人体动作识别综述[J]. 图学学报, 2023, 44(4): 625-639. |
[7] | 曹义亲, 周一纬, 徐露. 基于E-YOLOX的实时金属表面缺陷检测算法[J]. 图学学报, 2023, 44(4): 677-690. |
[8] | 邵俊棋, 钱文华, 徐启豪. 基于条件残差生成对抗网络的风景图生成[J]. 图学学报, 2023, 44(4): 710-717. |
[9] | 余伟群, 刘佳涛, 张亚萍. 融合注意力的拉普拉斯金字塔单目深度估计[J]. 图学学报, 2023, 44(4): 728-738. |
[10] | 郭印宏, 王立春, 李爽. 基于重复性和特异性约束的图像特征匹配[J]. 图学学报, 2023, 44(4): 739-746. |
[11] | 毛爱坤, 刘昕明, 陈文壮, 宋绍楼. 改进YOLOv5算法的变电站仪表目标检测方法[J]. 图学学报, 2023, 44(3): 448-455. |
[12] | 王佳婧, 王晨, 朱媛媛, 王笑梅. 基于民国纸币的图元素匹配检索[J]. 图学学报, 2023, 44(3): 492-501. |
[13] | 杨柳, 吴晓群. 基于深度学习的三维形状补全研究综述[J]. 图学学报, 2023, 44(2): 201-215. |
[14] | 曾武, 朱恒亮, 邢树礼, 林江宏, 毛国君. 显著性检测引导的图像数据增强方法[J]. 图学学报, 2023, 44(2): 260-270. |
[15] | 罗启明, 吴昊, 夏信, 袁国武. 基于Dual Dense U-Net的云南壁画破损区域预测[J]. 图学学报, 2023, 44(2): 304-312. |
阅读次数 | ||||||
全文 |
|
|||||
摘要 |
|
|||||