基于调色板的半交互式低照度唐卡图像增强

doi:10.11996/JG.j.2095-302X.2023061202

图学学报 ›› 2023, Vol. 44 ›› Issue (6): 1202-1211.DOI: 10.11996/JG.j.2095-302X.2023061202

• 图像处理与计算机视觉 • 上一篇下一篇

基于调色板的半交互式低照度唐卡图像增强

张驰¹^,²(), 张效娟¹^,²(), 赵洋³, 杨帆¹^,²

1.青海师范大学计算机学院，青海西宁 810016
2.省部共建藏语智能信息处理及应用国家重点实验室，青海西宁 810016
3.合肥工业大学计算机与信息学院，安徽合肥 230601

收稿日期:2023-06-20 接受日期:2023-09-22 出版日期:2023-12-31 发布日期:2023-12-17
通讯作者: 张效娟(1968-)，女，教授，硕士。主要研究方向为非物质文化遗产数字化保护、计算机视觉等。E-mail：zhxj@qhnu.edu.cn
作者简介:
张驰(1996-)，男，硕士研究生。主要研究方向为非物质文化遗产数字化保护、计算机视觉。E-mail：756629946@qq.com
基金资助:
青海省重点研发与转化计划项目(2021-GX-111);国家自然科学基金项目(62262056);国家重点研发计划重点专项(2020YFC1523300)

Palette-based semi-interactive low-light Thangka images enhancement

ZHANG Chi¹^,²(), ZHANG Xiao-juan¹^,²(), ZHAO Yang³, YANG Fan¹^,²

1. School of Computer, Qinghai Normal University, Xining Qinghai 810016, China
2. State Key Laboratory of Tibetan Intelligent Information Processing and Application, Xining Qinghai 810016, China
3. School of Computer and Information, Hefei University of Technology, Hefei Anhui 230601, China

Received:2023-06-20 Accepted:2023-09-22 Online:2023-12-31 Published:2023-12-17
Contact: ZHANG Xiao-juan (1968-), professor, master. Her main research interests cover digital protection of intangible cultural heritage, computer vision, etc. E-mail：zhxj@qhnu.edu.cn
About author:
ZHANG Chi (1996-), master student. His main research interests cover digital protection of intangible cultural heritage, computer vision. E-mail：756629946@qq.com
Supported by:
Key Research and Development and Transformation Plan Project in Qinghai Province(2021-GX-111);National Natural Science Foundation of China(62262056);Key Special Projects of National Key R&D Plan(2020YFC1523300)

摘要/Abstract

摘要：

唐卡作为热贡艺术重要表现形式之一，其结构复杂、颜色鲜艳、线条清晰、绘画精美，受到越来越多的人喜爱。在实际采集过程中，由于各个寺院灯光昏暗，在此条件下拍摄的图像常存在光照不均匀、噪声多、颜色失真、细节信息丢失等问题。因此提出了一种基于调色板的半交互式低照度唐卡图像增强方法：首先基于Retinex模型设计了一个卷积块注意力模块(CBAM)与U-Net相结合的低照度增强网络RCUNet，通过针对性设计损失函数，进行无监督迭代训练，对光照图、反射图和噪声图进行重构，并在得到最终分解结果后，对光照图、反射图进行调整，合成增强的结果。然后，采用改进的K-means算法，对增强后的图像提取主要颜色生成对应的调色板，通过修改调色板颜色进一步修正增强后图片的颜色。最后,与目前流行的几种低照度增强方法，在唐卡数据集上进行了定量、定性对比实验，实验结果表明该方法在NIQE，PIQE和PSNR 3个指标上取得了最好的结果。

关键词: 热贡唐卡, 低照度图像增强, Retinex模型, 注意力机制, 调色板

Abstract:

As one of the important expressions of Regong art, Thangka has garnered increasing popularity due to its complex structure, bright colors, clear lines, and exquisite paintings. However, capturing Thangka images in dimly lit temple settings often presents challenges such as uneven illumination, high noise, color distortion, and loss of detail information. To address these issues, a semi-interactive low illumination Thangka image enhancement method based on color palettes was proposed. Firstly, based on the Retinex model, a low-illumination enhancement network, RCUNet, incorporating the convolutional block attention module (CBAM) and U-Net, was designed. Through the designed loss function, iterative training was conducted to reconstruct the illumination, reflection, and noise maps, thus synthesizing an enhanced result. For interaction, the main colors of the enhanced image were extracted and corresponding color palettes were generated using an improved K-means algorithm. Then, modifying these color palettes further improved the colors of the enhanced image. Finally, compared with several currently popular enhancement methods, quantitative and qualitative comparison experiments were undertaken on the Thangka datasets. The experimental results demonstrated that this method could yield the best results in three indicators: NIQE, PIQE, and PSNR scores.

Key words: Regong Thangka, low-light image enhancement, Retinex model, attention mechanism, color palette

中图分类号:

TP391

张驰, 张效娟, 赵洋, 杨帆. 基于调色板的半交互式低照度唐卡图像增强[J]. 图学学报, 2023, 44(6): 1202-1211.

ZHANG Chi, ZHANG Xiao-juan, ZHAO Yang, YANG Fan. Palette-based semi-interactive low-light Thangka images enhancement[J]. Journal of Graphics, 2023, 44(6): 1202-1211.

图/表 13

图1 整体结构图

Fig. 1 Architecture of the proposed method

图2 唐卡图像((a)采集到的图像；(b)裁剪图像)

Fig. 2 Tangka images ((a) Collected images; (b) Crop images)

图3 分解网络结构图

Fig. 3 Architecture of the decomposition network

图4 反射重建网络结构图

Fig. 4 Architecture of the reflection reconstruction network

表1 在ExDark，LIME和无参考唐卡数据集上不同方法的NIQE和PIQE的结果

Table 1 Results of NIQE and PIQE using different methods on ExDark, LIME and unreferenced Thangka datasets

方法	ExDark		LIME		无参考唐卡数据集
方法	NIQE	PIQE	NIQE	PIQE	NIQE	PIQE
KinD	10.835 9	11.966 3	16.256 8	16.583 1	22.568 6	30.218 5
EnlightenGAN	10.320 1	10.365 6	15.239 8	10.494 5	20.760 9	16.625 2
Zero-DCE	11.322 2	12.789 5	13.407 4	11.640 7	19.401 4	12.369 1
RRDNet	9.961 5	11.750 6	13.763 7	10.796 2	18.351 6	12.337 7
RUAS	9.683 5	13.370 8	12.719 0	14.055 6	18.533 2	17.251 1
SCI	9.236 7	13.042 5	13.027 8	12.637 9	18.960 1	11.938 6
RCUNet	9.663 7	13.468 1	12.457 3	11.547 0	17.569 5	9.149 6

表2 在有参考唐卡数据集上不同方法的PSNR和SSIM的结果

Table 2 Results of PSNR and SSIM using different methods on a referenced Thangka datasets

方法	有参考唐卡数据集
方法	PSNR	SSIM
KinD	16.197 4	0.840 3
EnlightenGAN	17.585 5	0.865 5
Zero-DCE	18.238 2	0.852 0
RRDNet	17.892 5	0.846 4
RUAS	15.679 5	0.745 7
SCI	16.669 1	0.843 5
RCUNet	17.813 4	0.843 0
P-RCUNet	19.781 3	0.861 6

图5 可视化对比结果

Fig. 5 Visual comparison results ((a) Input; (b) KinD; (c) EnlightenGAN; (d) Zero-DCE; (e) RRDNet; (f) RUAS; (g) SCI; (h) RCUNet)

图6 有参考的唐卡数据集可视化对比结果

Fig. 6 Visual comparison results of referenced Thangka datasets ((a) Input; (b) KinD; (c) EnlightenGAN; (d) Zero-DCE; (e) RRDNet; (f) RUAS; (g) SCI; (h) RCUNet; (i) P-RCUNet)

图7 基于调色板的半交互式颜色修正实例图((a)输入图像；(b)真实图像；(c) RCUNet及其对应的调色板；(d) P-RCUNet及其对应的调色板)

Fig. 7 Example images of semi interactive color correction based on color palette ((a) Input; (b) Ground Truth; (c) RCUNet and corresponding color palette; (d) P-RCUNet and corresponding color palette)

图8 拼接后的可视化结果(局部)

Fig. 8 Visualization results after splicing (local) ((a) Input; (b) KinD; (c) EnlightenGAN; (d) Zero-DCE; (e) RRDNet; (f) RUAS; (g) SCI; (h) RCUNet; (i) P-RCUNet)

表3 唐卡低照度增强结果主观评价结果

Table 3 Subjective evaluation results of Thangka low illumination enhancement results

方法	实验组1	实验组2	实验组3	加权平均
KinD	3.22	3.19	3.12	3.151
EnlightenGAN	3.78	3.82	3.92	3.876
Zero-DCE	3.10	2.98	3.05	3.034
RRDNet	3.82	3.91	4.04	3.979
RUAS	3.31	3.27	3.19	3.226
SCI	3.24	3.16	3.17	3.174
RCUNet	3.82	3.99	3.98	3.967
P-RCUNet	3.92	4.06	4.11	4.076

表4 消融实验结果(无参考唐卡数据集)

Table 4 Results of ablation experiment (unreferenced Thangka dataset)

方法	无参考唐卡数据集
方法	NIQE	PIQE
RCUNet	17.569 5	9.149 6
w/o CBAM	21.203 5	10.573 9
w/o L_exp	21.490 0	11.750 6
w/o L_tv	21.253 1	10.817 3
w/o L_spa	20.358 7	9.895 7
w/o L_colo_r	20.997 4	10.282 2

表5 消融实验结果(有参考唐卡数据集)

Table 5 Results of ablation experiment (reference Tangka dataset)

方法	有参考唐卡数据集
方法	PSNR	SSIM
RCUNet	17.813 4	0.843 0
w/o CBAM	15.632 4	0.760 7
w/o L_exp	14.857 7	0.725 9
w/o L_tv	15.709 7	0.760 0
w/o L_spa	14.914 8	0.740 6
w/o L_color	15.920 7	0.771 1

参考文献 19

[1]	LAND E H. An alternative technique for the computation of the designator in the retinex theory of color vision[J]. Proceedings of the National Academy of Sciences of the United States of America, 1986, 83(10): 3078-3080. PMID
[2]	LORE K G, AKINTAYO A, SARKAR S. LLNet: a deep autoencoder approach to natural low-light image enhancement[J]. Pattern Recognition, 2017, 61: 650-662. DOI URL
[3]	JIANG Y F, GONG X Y, LIU D, et al. EnlightenGAN: deep light enhancement without paired supervision[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2021, 30: 2340-2349. DOI URL
[4]	GUO C L, LI C Y, GUO J C, et al. Zero-reference deep curve estimation for low-light image enhancement[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1777-1786.
[5]	WEI C, WANG W J, YANG W H, et al. Deep retinex decomposition for low-light enhancement[EB/OL]. [2023-02-13]. https://arxiv.org/abs/1808.04560.pdf.
[6]	ZHANG Y H, ZHANG J W, GUO X J. Kindling the darkness: a practical low-light image enhancer[C]// The 27th ACM International Conference on Multimedia. New York: ACM, 2019: 1632-1640.
[7]	ZHU A Q, ZHANG L, SHEN Y, et al. Zero-shot restoration of underexposed images via robust retinex decomposition[C]// 2020 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2020: 1-6.
[8]	LIU R S, MA L, ZHANG J A, et al. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 10556-10565.
[9]	MA L, MA T Y, LIU R S, et al. Toward fast, flexible, and robust low-light image enhancement[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 5627-5636.
[10]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[M]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 3-19.
[11]	RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[M]// Lecture Notes in Computer Science. Cham: Springer International Publishing, 2015: 234-241.
[12]	LI M D, LIU J Y, YANG W H, et al. Structure-revealing low-light image enhancement via robust retinex model[J]. IEEE Transactions on Image Processing, 2018, 27(6): 2828-2841. DOI PMID
[13]	CHANG H W, FRIED O, LIU Y M, et al. Palette-based photo recoloring[J]. ACM Transactions on Graphics, 2015, 34(4): 139:1-139:11.
[14]	LI C Y, GUO C L, HAN L H, et al. Low-light image and video enhancement using deep learning: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(12): 9396-9416. DOI URL
[15]	LEE C, LEE C, KIM C S. Contrast enhancement based on layered difference representation of 2D histograms[J]. IEEE Transactions on Image Processing, 2013, 22(12): 5372-5384. PMID
[16]	LOH Y P, CHAN C S. Getting to know low-light images with the exclusively dark dataset[J]. Computer Vision and Image Understanding, 2019, 178: 30-42. DOI
[17]	GUO X J, LI Y, LING H B. LIME: low-light image enhancement via illumination map estimation[J]. IEEE Transactions on Image Processing, 2017, 26(2): 982-993. DOI PMID
[18]	MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a “completely blind” image quality analyzer[J]. IEEE Signal Processing Letters, 2013, 20(3): 209-212. DOI URL
[19]	VENKATANATH N, PRANEETH D, BH M C, et al. Blind image quality evaluation using perception based features[C]// The 21st National Conference on Communications. New York: IEEE Press, 2015: 1-6.

基于调色板的半交互式低照度唐卡图像增强

Palette-based semi-interactive low-light Thangka images enhancement

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 19

相关文章 15

编辑推荐

Metrics

本文评价

[1]	郭宗洋, 刘立东, 蒋东华, 刘子翔, 朱熟康, 陈京华 . 基于语义引导神经网络的人体动作识别算法 [J]. 图学学报, 2024, 45(1): 26-34.
[2]	翟永杰, 赵晓瑜, 王璐瑶, 王亚茹, 宋晓轲, 朱浩硕. IDD-YOLOv7：一种用于输电线路绝缘子多缺陷的轻量化检测方法 [J]. 图学学报, 2024, 45(1): 90-101.
[3]	古天骏, 熊苏雅, 林晓. 基于SASGAN的戏剧脸谱多样化生成 [J]. 图学学报, 2024, 45(1): 102-111.
[4]	崔克彬, 焦静颐 . 基于MCB-FAH-YOLOv8的钢材表面缺陷检测算法 [J]. 图学学报, 2024, 45(1): 112-125.
[5]	魏陈浩, 杨睿, 刘振丙, 蓝如师, 孙希延, 罗笑南. 具有双层路由注意力的YOLOv8道路场景目标检测方法[J]. 图学学报, 2023, 44(6): 1104-1111.
[6]	丁建川, 肖金桐, 赵可新, 贾冬青, 崔炳德, 杨鑫. 基于脉冲神经网络的复杂场景导航避障算法[J]. 图学学报, 2023, 44(6): 1121-1129.
[7]	张丽媛, 赵海蓉, 何巍, 唐雄风. 融合全局-局部注意模块的Mask R-CNN膝关节囊肿检测方法[J]. 图学学报, 2023, 44(6): 1183-1190.
[8]	杨陈成, 董秀成, 侯兵, 张党成, 向贤明, 冯琪茗. 基于参考的Transformer纹理迁移深度图像超分辨率重建[J]. 图学学报, 2023, 44(5): 861-867.
[9]	宋焕生, 文雅, 孙士杰, 宋翔宇, 张朝阳, 李旭. 基于改进教师学生网络的隧道火灾检测[J]. 图学学报, 2023, 44(5): 978-987.
[10]	李利霞, 王鑫, 王军, 张又元. 基于特征融合与注意力机制的无人机图像小目标检测算法[J]. 图学学报, 2023, 44(4): 658-666.
[11]	李鑫, 普园媛, 赵征鹏, 徐丹, 钱文华. 内容语义和风格特征匹配一致的艺术风格迁移[J]. 图学学报, 2023, 44(4): 699-709.
[12]	余伟群, 刘佳涛, 张亚萍. 融合注意力的拉普拉斯金字塔单目深度估计[J]. 图学学报, 2023, 44(4): 728-738.
[13]	胡欣, 周运强, 肖剑, 杨杰. 基于改进YOLOv5的螺纹钢表面缺陷检测[J]. 图学学报, 2023, 44(3): 427-437.
[14]	郝鹏飞, 刘立群, 顾任远. YOLO-RD-Apple果园异源图像遮挡果实检测模型[J]. 图学学报, 2023, 44(3): 456-464.
[15]	罗文宇, 傅明月. 基于YoloX-ECA模型的非法野泳野钓现场监测技术[J]. 图学学报, 2023, 44(3): 465-472.