Prediction of damaged areas in Yunnan murals using Dual Dense U-Net

doi:10.11996/JG.j.2095-302X.2023020304

Abstract

Abstract:

The prediction of damaged areas of murals constitutes an important part of the virtual restoration of murals. However, current methods are prone to problems such as incomplete prediction of damaged areas and inaccurate prediction of damaged boundaries of the complex texture area in the case of Yunnan Minority Murals. To address these challenges, an improved Dual Dense U-Net segmentation model based on U-Net was proposed. This method enhanced the location and texture features of damaged regions, resulting in more discriminative information and the improved accuracy of damaged mask prediction. To enable the model to learn mural features more effectively, a segmentation dataset containing 5,000 images of Yunnan Minority Murals was established. The Dual Dense U-Net model employed a fusion module to perform a multi-scale fusion of mural images, mitigating the loss of local texture information and spatial position information in the feedforward process of mural images. First, the U-Net structure was used to extract information from the input mural image. The fusion module was comprised of multiple depthwise separable convolutions, which could improve the efficiency and segmentation accuracy of the fusion module. Secondly, the fusion module connected two U-Nets to further strengthen the connection between shallow features and deep features. The experimental results revealed that the IoU and Dice evaluation indicators of the model were improved by 3 percentage points compared with UNet++, and that the damaged areas predicted by the model could significantly improve the restoration effect of the murals restoration network. The proposed model was thus proven to be effective in predicting damaged areas of murals.

Key words: mural segmentation, disease extraction, depthwise separable convolution, multi-scale information fusion, deep learning

CLC Number:

TP391

LUO Qi-ming, WU Hao, XIA Xin, YUAN Guo-wu. Prediction of damaged areas in Yunnan murals using Dual Dense U-Net[J]. Journal of Graphics, 2023, 44(2): 304-312.

Figures/Tables 10

Fig. 1 Examples of damage types of murals (scratch in yellow frame, peeling in white frame, crack in green frame, fading in blue frame) ((a) Including scratches and peeling; (b) Including peeling, cracks and fading)

Fig. 2 Dual Dense U-Net model structure diagram

Fig. 3 Network module structure diagram

Fig. 4 Feedforward feature fusion module structure diagram

Fig. 5 Samples of YMIM

Table 1 Quantitative evaluation results of different models (%)

评估标准	分割模型
评估标准	U-Net	UNet+	UNet++	DeepLabV3+	Swin-Unet	GSCNN	MOoSe	HarDNet	DDU
IoU	44.85	45.32	47.98	32.11	43.11	48.49	40.43	45.51	50.99
Dice	59.76	60.11	63.08	47.27	58.51	63.91	55.51	60.65	66.27

Table 2 Ablation studies of different design choices (%)

D_conv	MLP	IoU	Dice
-	-	47.72	63.65
-	√	49.52	65.19
√	-	48.26	64.34
√	√	50.99	66.27

Fig. 6 Qualitative results of different models

Fig. 7 Comparison of different method masks in murals restoration applications ((a) The restoration result of large holes; (b) The restoration result of small damages)

Fig. 8 Ablation study results (“w/o MLP” means removing attention mechanism, “FULL” means full model) ((a), (b) Means ablation results of large holes; (c), (d) Means ablation results of small damages)

References 33

[1]	温利龙, 徐丹, 张熹, 等. 基于生成模型的古壁画非规则破损部分修复方法[J]. 图学学报, 2019, 40(5): 925-931.
	WEN L L, XU D, ZHANG X, et al. The inpainting of irregular damaged areas in ancient murals using generative model[J]. Journal of Graphics, 2019, 40(5): 925-931. (in Chinese)
[2]	PENG J L, LIU D, XU S C, et al. Generating diverse structure for image inpainting with hierarchical VQ-VAE[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 10770-10779.
[3]	DONG Q L, CAO C J, FU Y W. Incremental transformer structure enhanced image inpainting with masking positional encoding[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 11348-11358.
[4]	BEZDEK J C, EHRLICH R, FULL W. FCM: the fuzzy c-means clustering algorithm[J]. Computers & Geosciences, 1984, 10(2-3): 191-203. DOI URL
[5]	张子迎, 税午阳, 周明全, 等. 数字壁画病害提取与修复算法研究[J]. 计算机应用研究, 2021, 38(8): 2495-2498, 2504.
	ZHANG Z Y, SHUI W Y, ZHOU M Q, et al. Research on disease extraction and inpainting algorithm of digital grotto murals[J]. Application Research of Computers, 2021, 38(8): 2495-2498, 2504. (in Chinese)
[6]	吴萌, 王慧琴, 李文怡. 多尺度唐墓室壁画病害标记及修复技术研究[J]. 计算机工程与应用, 2016, 52(11): 169-174.
	WU M, WANG H Q, LI W Y. Research on multi-scale detection and image inpainting of Tang dynasty tomb murals[J]. Computer Engineering and Applications, 2016, 52(11): 169-174. (in Chinese)
[7]	JAIDILERT S, FAROOQUE G. Crack detection and images inpainting method for Thai mural painting images[C]// 2018 IEEE International Conference on Image, Vision and Computing. New York: IEEE Press, 2018: 143-148.
[8]	曹建芳, 田晓东, 贾一鸣, 等. 改进DeepLabV³⁺模型在壁画分割中的应用[J]. 计算机应用, 2021, 41(5): 1471-1476. DOI
	CAO J F, TIAN X D, JIA Y M, et al. Application of improved DeepLabV³⁺ model in mural segmentation[J]. Journal of Computer Applications, 2021, 41(5): 1471-1476. (in Chinese)
[9]	吕书强, 王诗涵, 侯妙乐, 等. 基于改进U-Net的壁画颜料层脱落病害区域提取[J]. 地理信息世界, 2022, 29(1): 69-74.
	LYU S Q, WANG S H, HOU M L, et al. Extraction of mural paint loss diseases based on improved U-net[J]. Geomatics World, 2022, 29(1): 69-74. (in Chinese)
[10]	RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[M]// Lecture Notes in Computer Science. Cham: Springer International Publishing, 2015: 234-241.
[11]	SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640-651. DOI PMID
[12]	LIN G S, MILAN A, SHEN C H, et al. RefineNet: multi-path refinement networks for high-resolution semantic segmentation[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5168-5177.
[13]	YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions[EB/OL]. [2022-07-11]. https://arxiv.org/abs/1511.07122.
[14]	HUANG Z L, WANG X G, HUANG L C, et al. CCNet: criss-cross attention for semantic segmentation[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2020: 603-612.
[15]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[EB/OL]. [2022-07-11]. https://arxiv.org/abs/1606.00915.
[16]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848. DOI URL
[17]	HE K M, ZHANG X Y, REN S Q, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916. DOI PMID
[18]	CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[EB/OL]. [2022-07-11]. https://arxiv.org/abs/1706.05587.
[19]	CHEN L C, YANG Y, WANG J, et al. Attention to scale: scale-aware semantic image segmentation[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 3640-3649.
[20]	MNIH V, HEESS N, GRAVES A, et al. Recurrent models of visual attention[EB/OL]. [2022-07-11]. https://arxiv.org/abs/1406.6247.
[21]	LEE S, CHOI W, KIM C, et al. ADAS: a direct adaptation strategy for multi-target domain adaptive semantic segmentation[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 19174-19184.
[22]	CHENG Y T, WEI F Y, BAO J M, et al. Dual path learning for domain adaptation of semantic segmentation[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2022: 9062-9071.
[23]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[M]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 3-19.
[24]	SANDLER M, HOWARD A, ZHU M L, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 4510-4520.
[25]	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1800-1807.
[26]	HARTIGAN J A, WONG M A. Algorithm AS 136: a K-means clustering algorithm[J]. Applied Statistics, 1979, 28(1): 100. DOI URL
[27]	ZHOU Z W, SIDDIQUEE M M R, TAJBAKHSH N, et al. UNet: redesigning skip connections to exploit multiscale features in image segmentation[J]. IEEE Transactions on Medical Imaging, 2020, 39(6): 1856-1867. DOI URL
[28]	CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[M]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 833-851.
[29]	CAO H, WANG Y Y, CHEN J, et al. Swin-unet: unet-like pure transformer for medical image segmentation[EB/OL]. [2022-07-11]. https://arxiv.org/abs/2105.05537.
[30]	TAKIKAWA T, ACUNA D, JAMPANI V, et al. Gated-SCNN: gated shape CNNs for semantic segmentation[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2020: 5228-5237.
[31]	GALESSO S, BRAVO M A, NAOUAR M, et al. Probing contextual diversity for dense out-of-distribution detection[EB/OL]. [2022-07-11]. https://arxiv.org/abs/2208.14195.
[32]	LIAO T Y, YANG C H, LO Y W, et al. HarDNet-DFUS: an enhanced harmonically-connected network for diabetic foot ulcer image segmentation and colonoscopy polyp segmentation[EB/OL]. [2022-07-11]. https://arxiv.org/abs/2209.07313.
[33]	LEMPITSKY V, VEDALDI A, ULYANOV D. Deep image prior[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 9446-9454.