Automatic portrait matting model based on semantic guidance

doi:10.11996/JG.j.2095-302X.2024040683

Abstract

Abstract:

To address the issues of semantic discrimination errors and unclear details in existing portrait matting methods, an automatic matting model based on semantic guidance was proposed.Firstly, a hybrid CNN-Transformer architecture EMO was introduced for feature encoding. Then, the semantic segmentation decoding branch utilized a multi-scale hybrid attention module to process the top-level encoded features, enhancing multi-scale representation and pixel-level discrimination capabilities. Next, a feature enhancement module was employed to merge high-level features, facilitating the flow of high-level semantic information through the shallow network. Simultaneously, the aggregation guidance module in the detail extraction decoding branch aggregated features from different branches, utilizing the aggregated features to better guide the network in extracting shallow features, thereby improving the accuracy of edge and detail extraction. Experiments on three datasets demonstrated that our approach outperformed the compared methods, achieving optimal performance while significantly reducing parameter count and computational complexity, validating the competitiveness of our proposed method.

Key words: portrait matting, semantic guidance, multi-scale, feature enhancement, aggregation guidance

CLC Number:

TP391

CHENG Yan, YAN Zhihang, LAI Jianming, WANG Guixi, ZHONG Linhui. Automatic portrait matting model based on semantic guidance[J]. Journal of Graphics, 2024, 45(4): 683-695.

Figures/Tables 22

References 47

[1]	ZHANG J, TAO D C. Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things[J]. IEEE Internet of Things Journal, 2021, 8(10): 7789-7817.
[2]	CHEN Q F, LI D, TANG C K. KNN matting[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(9): 2175-2188. DOI PMID
[3]	PORTER T, DUFF T. Compositing digital images[C]// The 11th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1984: 253-259.
[4]	AKSOY Y, AYDIN T O, POLLEFEYS M. Designing effective inter-pixel information flow for natural image matting[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 228-236.
[5]	CHUANG Y Y, CURLESS B, SALESIN D H, et al. A Bayesian approach to digital matting[C]// 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Computer Society, 2001: 264.
[6]	GASTAL E S L, OLIVEIRA M M. Shared sampling for real-time alpha matting[J]. Computer Graphics Forum, 2010, 29(2): 575-584.
[7]	LEVIN A, LISCHINSKI D, WEISS Y. A closed-form solution to natural image matting[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(2): 228-242. PMID
[8]	LEVIN A, RAV-ACHA A, LISCHINSKI D. Spectral matting[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(10): 1699-1712. DOI PMID
[9]	SUN J, JIA J Y, TANG C K, et al. Poisson matting[J]. ACM Transactions on Graphics, 23(3): 315-321.
[10]	PHAM V Q, TAKAHASHI K, NAEMURA T. Real-time video matting based on bilayer segmentation[C]// Asian Conference on Computer Vision. Berlin, Heidelberg: Springer, 2010: 489-501.
[11]	XU N, PRICE B, COHEN S, et al. Deep image matting[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 311-320.
[12]	LIU Y H, XIE J K, SHI X, et al. Tripartite information mining and integration for image matting[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 7535-7544.
[13]	SENGUPTA S, JAYARAM V, CURLESS B, et al. Background matting: the world is your green screen[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 2288-2297.
[14]	LIN S C, YANG L J, SALEEMI I, et al. Robust high-resolution video matting with temporal guidance[C]// 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2022: 3132-3141.
[15]	彭泓, 张家宝, 贾迪, 等. 结合背景图的高分辨率视频人像实时抠图网络[J]. 中国图象图形学报, 2024, 29(2): 478-490.
	PENG H, ZHANG J B, JIA D, et al. Real-time high-resolution video portrait matting network combined with background image[J]. Journal of Image and Graphics, 2024, 29(2): 478-490 (in Chinese).
[16]	CHEN Q, GE T Z, XU Y Y, et al. Semantic human matting[C]// The 26th ACM International Conference on Multimedia. New York: ACM, 2018: 618-626.
[17]	DEORA R, SHARMA R, RAJ D S S. Salient image matting[EB/OL]. [2023-10-19]. http://arxiv.org/abs/2103.12337.
[18]	SHARMA R, DEORA R, VISHVAKARMA A. AlphaNet: an attention guided deep network for automatic image matting[C]// 2020 International Conference on Omni-layer Intelligent Systems. New York: IEEE Press, 2020: 1-8.
[19]	ZHANG Y K, GONG L X, FAN L B, et al. A late fusion CNN for digital matting[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 7461-7470.
[20]	QIAO Y, LIU Y H, YANG X, et al. Attention-guided hierarchical structure aggregation for image matting[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 13673-13682.
[21]	LI J, ZHANG J, MAYBANK S J, et al. Bridging composite and real: towards end-to-end deep image matting[J]. International Journal of Computer Vision, 2022, 130(2): 246-266.
[22]	LI J, ZHANG J, TAO D C. Deep automatic natural image matting[EB/OL]. [2023-10-19]. https://arxiv.org/abs/2107.07235.
[23]	苏常保, 龚世才. 基于深度学习的人物肖像全自动抠图算法[J]. 图学学报, 2022, 43(2): 247-253.
	SU C B, GONG S C. Fully automatic matting algorithm for portraits based on deep learning[J]. Journal of Graphics, 2022, 43(2): 247-253 (in Chinese). DOI
[24]	KE Z H, SUN J Y, LI K C, et al. MODNet: real-time trimap-free portrait matting via objective decomposition[C]// 2022 AAAI Conference on Artificial Intelligence. Vancouver: AAAI Press, 2022: 1140-1147.
[25]	MA S H, LI J, ZHANG J, et al. Rethinking portrait matting with privacy preserving[J]. International Journal of Computer Vision, 2023, 131(8): 2172-2197.
[26]	O’SHEA K, NASH R. An introduction to convolutional neural networks[EB/OL]. [2023-10-19]. http://arxiv.org/abs/1511.08458.
[27]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// The 31st International Conference on Neural Information Processing Systems, Red Hook: NIPS, 2017: 6000-6010.
[28]	ZHANG J N, LI X T, LI J, et al. Rethinking mobile block for efficient attention-based models[EB/OL]. [2023-10-19]. http://arxiv.org/abs/2301.01146.
[29]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 770-778.
[30]	HOU Q B, ZHANG L, CHENG M M, et al. Strip pooling: rethinking spatial pooling for scene parsing[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 4002-4011.
[31]	HUANG Z L, WANG X G, WEI Y C, et al. CCNet: criss-cross attention for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(6): 6896-6908.
[32]	FU J, LIU J, TIAN H J, et al. Dual attention network for scene segmentation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 3141-3149.
[33]	GUO M H, LU C Z, HOU Q B, et al. SegNeXt: rethinking convolutional attention design for semantic segmentation[EB/OL]. [2023-10-19]. http://arxiv.org/abs/2209.08575.
[34]	TAKIKAWA T, ACUNA D, JAMPANI V, et al. Gated-SCNN: gated shape CNNs for semantic segmentation[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 5228-5237.
[35]	HOU Q Q, LIU F. Context-aware image matting for simultaneous foreground and alpha estimation[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 4129-4138.
[36]	KINGMA D, BA J. Adam: a method for stochastic optimization[C]// 2015 International Conference on Learning Representations. San Diego: ICLR, 2015: 1-15.
[37]	YU Q H, ZHANG J M, ZHANG H, et al. Mask guided matting via progressive refinement network[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 1154-1163.
[38]	RHEMANN C, ROTHER C, WANG J, et al. A perceptually motivated online benchmark for image matting[C]// 2009 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2009: 1826-1833.
[39]	HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 2261-2269.
[40]	XIE S N, GIRSHICK R, DOLLÁR P, et al. Aggregated residual transformations for deep neural networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5987-5995.
[41]	SANDLER M, HOWARD A, ZHU M L, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 4510-4520.
[42]	LIU Z, LIN Y T, CAO Y, et al. Swin Transformer: hierarchical Vision Transformer using Shifted Windows[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 9992-10002.
[43]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]// European Conference on Computer Vision. Cham: Springer, 2018: 3-19.
[44]	OUYANG D L, HE S, ZHANG G Z, et al. Efficient multi-scale attention module with cross-spatial learning[C]// 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. New York: IEEE Press, 2023: 1-5.
[45]	PAN X R, GE C J, LU R, et al. On the integration of self-attention and convolution[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 805-815.
[46]	GE C J, DING X H, TONG Z, et al. Advancing vision transformers with group-mix attention[EB/OL]. [2023-10-19]. http://arxiv.org/abs/2311.15157.
[47]	MEHTA S, RASTEGARI M. MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer[EB/OL]. [2023-10-19]. http://arxiv.org/abs/2110.02178.

编码器变体	各阶段iRMB 堆叠次数	通道大小	参数量/M
EMO_T	(2，2，8，3)	(32，48，80，168)	1.3
EMO_S	(3，3，9，3)	(32，48，120，200)	2.3
EMO_B	(3，3，9，3)	(48，72，160，288)	5.1

编码器变体	各阶段iRMB 堆叠次数	通道大小	参数量/M
EMO_T	(2，2，8，3)	(32，48，80，168)	1.3
EMO_S	(3，3，9，3)	(32，48，120，200)	2.3
EMO_B	(3，3，9，3)	(48，72，160，288)	5.1

方法	骨干网络	SAD↓	MSE↓	MAD↓	Grad↓	Conn↓
LFM	DenseNet-201	31.65	12.76	16.81	30.29	18.74
HATT	ResNeXt-101	25.97	7.21	15.42	25.29	14.91
SHM	ResNet-50	21.95	9.93	12.76	18.17	27.06
GFM	ResNet-34	13.09	5.12	7.68	17.34	12.63
MODNet	MobileNetV2	12.93	4.57	7.54	13.31	12.38
P3M	ResNet-34	10.29	3.64	5.98	13.69	10.92
P3M-Swin	Swin	9.60	3.47	5.72	13.21	9.13
APM-SG(T)	EMO_T	9.49	3.34	5.56	13.17	9.26
APM-SG(S)	EMO_S	8.56	2.78	4.92	12.49	8.09
APM-SG(B)	EMO_B	8.09	2.59	4.63	12.10	7.68

方法	骨干网络	SAD↓	MSE↓	MAD↓	Grad↓	Conn↓
LFM	DenseNet-201	31.65	12.76	16.81	30.29	18.74
HATT	ResNeXt-101	25.97	7.21	15.42	25.29	14.91
SHM	ResNet-50	21.95	9.93	12.76	18.17	27.06
GFM	ResNet-34	13.09	5.12	7.68	17.34	12.63
MODNet	MobileNetV2	12.93	4.57	7.54	13.31	12.38
P3M	ResNet-34	10.29	3.64	5.98	13.69	10.92
P3M-Swin	Swin	9.60	3.47	5.72	13.21	9.13
APM-SG(T)	EMO_T	9.49	3.34	5.56	13.17	9.26
APM-SG(S)	EMO_S	8.56	2.78	4.92	12.49	8.09
APM-SG(B)	EMO_B	8.09	2.59	4.63	12.10	7.68

方法	骨干网络	SAD↓	MSE↓	MAD↓	Grad↓	Conn↓
LFM	DenseNet-201	40.71	16.34	23.72	41.36	17.63
HATT	ResNeXt-101	30.53	9.18	17.63	27.42	19.88
SHM	ResNet-50	23.63	10.61	13.68	15.17	28.52
GFM	ResNet-34	14.98	9.04	8.65	17.57	14.68
MODNet	MobileNetV2	15.68	6.14	9.23	13.63	15.29
P3M	ResNet-34	12.88	4.63	7.42	12.85	12.31
P3M-Swin	Swin	9.84	3.24	5.74	11.45	9.43
APM-SG(T)	EMO_T	9.69	3.13	5.65	11.20	9.25
APM-SG(S)	EMO_S	8.77	2.65	5.13	10.82	8.32
APM-SG(B)	EMO_B	8.62	2.49	5.02	10.67	8.18