TCPColor: a Chinese painting color scheme recommendation system based on text-to-image generation model

doi:10.11996/JG.j.2095-302X.2025030520

Abstract

Abstract:

Traditional Chinese painting (TCP) is a unique form of painting specific to China. Exploring the use of color schemes based on TCP holds significant importance for modern designers in integrating traditional art with contemporary design concepts. However, limited research was conducted on color recommendation systems based on TCP knowledge, and no effective solutions have yet been provided for color scheme retrieval and recommendation based on multi-dimensional features such as themes, objects, and artistic conceptions. A traditional Chinese painting color scheme recommendation system named TCPColor was proposed. Based on the Taiyi Chinese text-to-image generation model, this system fine-tuned the model using Song Dynasty TCP data annotated by experts. Then, it employed visual saliency algorithms, K-Means clustering, and color-distance-based palette matching on the generated images to produce color schemes that reflected the style of traditional Chinese painting. The effectiveness of the color extraction method was verified through ablation experiments, while objective color analysis was used to evaluate the distinctiveness of the generated color schemes and their similarity to traditional Chinese painting color schemes. In collaboration with TCP experts and volunteers, case studies, expert evaluations, and user research were conducted, which demonstrated the practicality of the system in recommending color schemes.

Key words: traditional Chinese painting, color scheme, color recommendation system, word-color association, text- to-image generation

CLC Number:

TP391
J212

ZHANG Di, ZHANG Wenan, JIANG Zhide, WU Aixia, KONG Hao, GUO Xian, CHEN Wei. TCPColor: a Chinese painting color scheme recommendation system based on text-to-image generation model[J]. Journal of Graphics, 2025, 46(3): 520-531.

Figures/Tables 14

Fig. 1 Overall system framework

Table 1 Rules of the Chinese painting annotation system

特征类别	特征描述	标签举例
主题	国画的画科	山水画、人物画、花鸟画
意境	画作在情感、氛围、思想、哲理等方面所展现出的深远意义	韶秀、沉雄、空灵等^[32]
技法	画作中使用的具有代表性的技法	披麻皴、双勾法等
物象	画作中包含的物象	叶、鸟、花等
构图	画作的构图特点	之字形构图，两线构图等
赋彩	画作的用色方案	设色、水墨、白描等

Table 2 Fine-tuning of the TGM model

步骤	参数/技术	详细说明
数据集构建	数据来源	故宫博物院开放数据专区
数据集构建	预处理步骤	去除图片背景，图片对应的标签集合并为描述语句
模型微调参数设置	硬件设备	2台Nvidia 3090显卡
	迭代次数	10个epoch
	学习率	32
	精度	fp16 (半精度)

Fig. 2 Comparison of generation results from different models using Chinese painting-related prompt words ((a) Flower and bird painting, bird, outline and wash technique; (b) Figure painting, maid, exquisite and divine; (c) Figure painting, monk; (d) Average score)

Table 3 Results of ablation study

步骤	评价标准	完整处理	消融处理
前景图像提取	CR	0.9	0.7
K-Means颜色聚类	用户满意度	3.9	2.7
国画标准色替代	CH	0.7	0.5

Fig. 3 Ablation study comparison results

Fig. 4 Objective color analysis of color schemes for various models ((a) Assessment of color differences; (b) Evaluation of similarity to traditional Chinese painting color schemes)

Fig. 5 Generation results of recommended color schemes((a) Generated image color scheme; (b) Recommended color scheme)

Fig. 6 System visual interface ((a) Control panel; (b) Overview of the painting; (c) Recommended color scheme; (d) Test color scheme)

Fig. 7 Case 1: distilling the beauty of national colors-the design process of Chinese-style product packaging color schemes ((a) Case prompt words; (b) Overview of the painting; (c) Recommended color scheme; (d) Color effect diagram for packaging box)

Fig. 8 Original image color scheme

Fig. 9 Case 3: recommending monochromatic schemes for Chinese painting ((a) Recommended color scheme for case 3; (b) Color effect diagram of fabric with cloud pattern)

Fig. 10 Comparison with other color schemes

Fig. 11 User experiment

References 37

[1]	陈佳舟, 黄可妤, 封颖超杰, 等. 基于古代绘画的古诗自动生成方法[J]. 计算机辅助设计与图形学学报, 2021, 33(7): 1038-1044.
	CHEN J Z, HUANG K Y, FENG Y C J, et al. Automatic poetry generation based on ancient Chinese paintings[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(7): 1038-1044 (in Chinese).
[2]	LIANG K. Color performance of Chinese flower-and-bird fine brushwork[EB/OL]. [2024-05-05]https://www.atlantis-press.com/proceedings/icesame-16/25850996.
[3]	TIAN Z X. Chinese style digital painting[D]. Menomonie: University of Wisconsin-Stout, 2017.
[4]	YU T X, LIN C, ZHANG S J, et al. End-to-end partial convolutions neural networks for Dunhuang grottoes wall-painting restoration[C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop. New York: IEEE Press, 2019: 1447-1455.
[5]	XU L, PARK S J, LEE S. Color2Vec: web-based modeling of word-color association with sociocultural contexts[J]. ACM Transactions on Computer-Human Interaction, 2023, 30(4): 51.
[6]	PERCHOULI AI. 中国色[EB/OL]. [2024-05-05]https://www.zhongguose.com/.
	PERCHOULI AI. Chinese colors[EB/OL]. [2024-05-05]https://www.zhongguose.com/.
[7]	MCCARTHY B, GIACCAI J. Scientific studies of pigments in Chinese paintings[M]. London: Archetype Publications, 2021: 158.
[8]	LIZUN D, KURKIEWICZ T, SZCZUPAK B, et al. A multi-analytical investigation of Liu Kang’s colour palette and painting technique from the Shanghai period (1933-1937)[J]. Applied Sciences, 2023, 13(4): 2414.
[9]	CHEN Y, YANG J, PAN Q Q, et al. A method for exploring word-colour associations[J]. Color Research & Application, 2020, 45(1): 85-94.
[10]	HU J Y, HAN X, YI X Y, et al. Efficient cross-lingual transfer for Chinese stable diffusion with images as pivots[EB/OL]. [2024-05-05]https://arxiv.org/abs/2305.11540.
[11]	O’DONOVAN P, AGARWALA A, HERTZMANN A. Color compatibility from large datasets[C]// ACM SIGGRAPH 2011 Papers. New York: ACM, 2011: 63.
[12]	MUNSELL A H. A color notation[M]. New York: Munsell Color Co., 1919: 1-146.
[13]	GRANVILLE W C. The color harmony manual, a color atlas based on the Ostwald color system[J]. Color Research & Application, 1994, 19(2): 77-98.
[14]	KOBAYASHI S. The aim and method of the color image scale[J]. Color Research & Application, 1981, 6(2): 93-107.
[15]	WANG W N, YU Y L. Image emotional semantic query based on color semantic description[C]// 2005 International Conference on Machine Learning and Cybernetics. New York: IEEE Press, 2005: 4571-4576.
[16]	CHEN T W, CHEN Y L, CHIEN S Y. Fast image segmentation based on K-means clustering with histograms in HSV color space[C]// 2008 IEEE 10th Workshop on Multimedia Signal Processing. New York: IEEE Press, 2008: 322-325.
[17]	DELON J, DESOLNEUX A, LISANI J L, et al. Automatic color palette[C]// IEEE International Conference on Image Processing 2005. New York: IEEE Press, 2005: II-706.
[18]	ACHANTA R, HEMAMI S, ESTRADA F, et al. Frequency-tuned salient region detection[C]// 2009 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2009: 1597-1604.
[19]	刘静, 庄梅玲, 商蕾, 等. 基于色彩聚类的皮影服饰纹样分割[J]. 现代纺织技术, 2021, 29(5): 71-77.
	LIU J, ZHUANG M L, SHANG L, et al. Pattern segmentation of shadow puppetry costumes based on color clustering[J]. Advanced Textile Technology, 2021, 29(5): 71-77 (in Chinese).
[20]	PHAN H Q, FU H B, CHAN A B. Color orchestra: ordering color palettes for interpolation and prediction[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(6): 1942-1955.
[21]	HAN J, LEE Y. Image sentiment considering color palette recommendations based on influence scores for image advertisement[J]. Electronic Commerce Research, 2024: 1-28.
[22]	HONG Y, ZENG X Y, WANG Y Y, et al. CBCRS: an open case-based color recommendation system[J]. Knowledge- Based Systems, 2018, 141: 113-128.
[23]	LEE H, ULLAH U, LEE J S, et al. A brief survey of text driven image generation and maniulation[C]// 2021 IEEE International Conference on Consumer Electronics-Asia. New York: IEEE Press, 2021: 1-4.
[24]	ROMBACH R, BLATTMANN A, LORENZ D, et al. High-resolution image synthesis with latent diffusion models[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 10674-10685.
[25]	RAMESH A, DHARIWAL P, NICHOL A, et al. Hierarchical text-conditional image generation with CLIP latents[EB/OL]. [2024-05-05]https://arxiv.org/abs/2204.06125.
[26]	KO H K, PARK G, JEON H, et al. Large-scale text-to-image generation models for visual artists’ creative works[C]// The 28th International Conference on Intelligent User Interfaces. New York: ACM, 2023: 919-933.
[27]	GAL R, ALALUF Y, ATZMON Y, et al. An image is worth one word: personalizing text-to-image generation using textual inversion[EB/OL]. [2024-05-05]https://arxiv.org/abs/2208.01618.
[28]	RUIZ N, LI Y Z, JAMPANI V, et al. Dreambooth: fine tuning text-to-image diffusion models for subject-driven generation[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 22500-22510.
[29]	ZHANG J X, GAN R Y, WANG J J, et al. Fengshenbang 1.0: being the foundation of Chinese cognitive intelligence[EB/OL]. [2024-05-05]https://arxiv.org/abs/2209.02970#.
[30]	LIU V, CHILTON L B. Design guidelines for prompt engineering text-to-image generative models[C]// 2022 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2022: 384.
[31]	XU H R, CHEN S Y, ZHANG Y. Magical brush: a symbol-based modern Chinese painting system for novices[C]// 2023 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2023: 131.
[32]	National Palace Museum. Open data[EB/OL].[2024-05-05]https://digitalarchive.npm.gov.tw/.opendata.
[33]	吴功正. 我写《六朝美学史》[J]. 中国美术, 1995(5): 5-7.
	WU Q Z. I wrote history of aesthetics in the Six Dynasties[J]. Art in China, 1995(5):5-7.
[34]	LERTRUSDACHAKUL T, RUXPAITOON K, THIPTARAJAN K. Color palette extraction by using modified k-means clustering[C]// 2019 7th International Electrical Engineering Congress. New York: IEEE Press, 2019: 1-4.
[35]	邝智杰. 基于颜色和谐度的可计算图像美学评估关键技术研究与应用[D]. 北京: 北京邮电大学, 2015.
	KUANG Z J. Color harmony based computational a esthetics of image and its application[D]. Beijing: Beijing University of Posts and Telecommunications, 2015 (in Chinese).
[36]	INOUE K, JIANG M Y, HARA K. Hue-preserving saturation improvement in RGB color cube[J]. Journal of Imaging, 2021, 7(8): 150.
[37]	ZHANG L M, LI C Z, SIMO-SERRA E, et al. User-guided line art flat filling with split filling mechanism[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 9884-9893.