Image colorization via semantic similarity propagation

doi:10.11996/JG.j.2095-302X.2025010126

Abstract

Abstract:

Image colorization aims to convert grayscale images into color images, a technique that has long received extensive attention from researchers in the fields of computer graphics and computer vision. It has found wide applications in areas such as image restoration, medical imaging, film restoration, and artistic creation. Over decades of development, researchers have proposed a large number of interaction-based, rule-based, and deep learning-based algorithms to enhance the colorization effect of images. Nevertheless, the existing image colorization algorithms exhibit some significant shortcomings, such as low computational efficiency, cumbersome user interaction, low color saturation, and the occurrence of color overflow. To address these challenges, an image colorization algorithm based on semantic similarity propagation was proposed. Semantic features of the input grayscale image were extracted using deep neural networks, and a feature space was constructed. Then, the image colorization task was formalized as an efficient energy optimization problem based on semantic similarity propagation, enabling the propagation of user-supplied stroke colors to other regions of the image. In addition, a trilinear interpolation method was employed to accelerate both energy optimization and color propagation, significantly enhancing computational efficiency. In order to verify the effectiveness of the algorithm, experiments were conducted on a collected image set, evaluating multiple dimensions, such as image visual effect, generated image quality, algorithm running time, and user interaction experience. The results of a large number of qualitative and quantitative experiments demonstrated that the proposed algorithm achieved more accurate, efficient, and natural colorization with reduced user interaction requirements, while achieving substantial improvements in computational efficiency.

Key words: image colorization, interactive, semantic similarity, propagation, energy optimization

CLC Number:

TP391.41

MENG Sihong, LIU Hao, FANG Haotian, SENG Bingfeng, DU Zhengjun. Image colorization via semantic similarity propagation[J]. Journal of Graphics, 2025, 46(1): 126-138.

Figures/Tables 13

Fig. 1 Overview of the algorithm

Fig. 2 Comparison of colorization results before and after optimization and before and after adding lightness ((a) Input image and strokes; (b) Initial semantic visualization results; (c) Results by combining the semantics and brightness of (b); (d) Optimized semantic visualization results; (e) Results obtained by combining the semantics and brightness of (d); (f) Results based only on the semantics of (d))

Fig. 3 Colorization result ((a) Bedroom; (b) Back view; (c) Pastoral)

Fig. 4 The effect of the choice of dimensions on our colorization results and running time ((a) Input image and strokes; (b) 2 dimension; (c) 3 dimensions; (d) 5 dimensions)

Fig. 5 Effect of parameter b on our colorization results and running time ((a) Input images and strokes; (b) b=2; (c) b=4; (d) b=8)

Fig. 6 Comparison of colorization results and running time of ours with other edit propagation algorithms ((a) Input images and strokes; (b) Reference [1]; (c) Ours)

Fig. 7 Comparison of our algorithm with other stroke-based deep learning algorithms ((a) Input images and strokes; (b) Reference [23]; (d) Reference [33]; (d) Reference [25]; (e) Ours)

Fig. 8 Comparison of our algorithm with other deep learning algorithms ((a) Input images; (b) Reference [15]; (c) Reference [24]; (d) Reference [23]; (e) Ours)

Fig. 9 Quantitative comparison of our algorithm with other colorization algorithms ((a) Input images; (b) Reference [15]; (c) Reference [25]; (d) Reference [24]; (e) Ours)

Fig. 10 Ablation study to verify the effectiveness of semantic feature ((a) Input images; (b) Luminance only; (c) Semantics only; (d) Semantics and luminance)

Table 1 Running time before and after accelerating

分辨率	示例	加速前/s	加速后/s
90×60	卧室	19.14	0.02
	背影	18.97	0.03
	田园	17.82	0.03
	树屋	19.64	0.03
	海边	18.74	0.03
	平均	18.86	0.03
128×85	卧室	150.89	0.03
	背影	143.87	0.02
	田园	155.31	0.03
	树屋	154.87	0.03
	海边	165.28	0.02
	平均	154.04	0.03
180×120	卧室	1207.19	0.04
	背影	1195.92	0.05
	田园	1183.51	0.04
	树屋	1164.31	0.05
	海边	1169.94	0.04
	平均	1184.17	0.04
256×171	卧室	10398.67	0.10
	背影	10274.71	0.07
	田园	10904.38	0.09
	树屋	10566.61	0.12
	海边	10568.34	0.09
	平均	10542.54	0.09

Table 2 Running time of all examples

示例	图像分辨率	运行时间/s
卧室(图3)	1280×853	0.58
背影(图3)	1280×853	0.53
田园(图3)	1280×853	0.40
小猫(图5)	300×199	0.03
牛群(图5)	1280×853	0.49
玫瑰(图6)	427×285	0.06
核桃(图6)	400×276	0.05
香蕉(图7)	640×960	0.24
花朵(图7)	853×1280	0.39
饼干(图8)	848×1280	0.44
陶瓷(图8)	1280×850	0.41
胶囊(图8)	1280×853	0.38
树屋(图9)	1280×853	0.44
月亮(图9)	853×1280	0.39
海边(图9)	1280×853	0.42
田野(图10)	1280×853	0.49
黄花(图10)	853×1280	0.40
平均		0.34

Fig. 11 Failure case ((a) Input image; (b) Initial semantics; (c) Optimized semantics; (d) Colorization results)

References 36

[1]	LEVIN A, LISCHINSKI D, WEISS Y. Colorization using optimization[C]// ACM SIGGRAPH 2004 Papers. New York: ACM, 2004: 689-694.
[2]	HUANG Y C, TUNG Y S, CHEN J C, et al. An adaptive edge detection based colorization algorithm and its applications[C]// The 13th Annual ACM International Conference on Multimedia. New York: ACM, 2005: 351-354.
[3]	YATZIV L, SAPIRO G. Fast image and video colorization using chrominance blending[J]. IEEE Transactions on Image Processing, 2006, 15(5): 1120-1129. PMID
[4]	HEU J H, HYUN D Y, KIM C S, et al. Image and video colorization based on prioritized source propagation[C]// The 2009 16th IEEE International Conference on Image Processing. New York: IEEE Press, 2009: 465-468.
[5]	QU Y G, WONG T T, HENG P A. Manga colorization[J]. ACM Transactions on Graphics (TOG), 2006, 25(3): 1214-1220.
[6]	LUAN Q, WEN F, COHEN-OR D, et al. Natural image colorization[C]// The 18th Eurographics conference on Rendering Techniques. Goslar: Eurographics Association, 2007: 309-320.
[7]	王泽文, 张小明. 基于p-Laplace方程的图像彩色化方法[J]. 工程图学学报, 2010, 31(6): 62-67.
	WANG Z W, ZHANG X M. Image colorization based on p-Laplace equation[J]. Journal of Engineering Graphics, 2010, 31(6): 62-67 (in Chinese).
[8]	田建勇, 石林江. 局部线性嵌入与模糊C-均值聚类的红外图像彩色化算法[J]. 图学学报, 2018, 39(5): 917-925. DOI
	TIAN J Y, SHI L J. An infrared image colorization algorithm based on local linear embedding and fuzzy C-means clustering[J]. Journal of Graphics, 2018, 39(5): 917-925 (in Chinese).
[9]	REINHARD E, ADHIKHMIN M, GOOCH B, et al. Color transfer between images[J]. IEEE Computer Graphics and Applications, 2001, 21(5): 34-41.
[10]	WELSH T, ASHIKHMIN M, MUELLER K. Transferring color to greyscale images[C]// The 29th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 2002: 277-280.
[11]	IRONY R, COHEN-OR D, LISCHINSKI D. Colorization by example[C]// The 16th Eurographics Conference on Rendering Techniques. Goslar: Eurographics Association, 2005: 201-210.
[12]	CHIA A Y S, ZHUO S J, GUPTA R K, et al. Semantic colorization with internet images[J]. ACM Transactions on Graphics (TOG), 2011, 30(6): 1-8.
[13]	GUPTA R K, CHIA A Y S, RAJAN D, et al. Image colorization using similar images[C]// The 20th ACM International Conference on Multimedia. New York: ACM, 2012: 369-378.
[14]	CHENG Z Z, YANG Q X, SHENG B. Deep colorization[C]// 2015 IEEE International Conference on Computer Vision. New York: IEEE Press, 2015: 415-423.
[15]	ZHANG R, ISOLA P, EFROS A A. Colorful image colorization[C]// The 14th European Conference on Computer Vision. Cham: Springer, 2016: 649-666.
[16]	ZHAO J J, HAN J G, SHAO L, et al. Pixelated semantic colorization[J]. International Journal of Computer Vision, 2020, 128(4): 818-834.
[17]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[18]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// The 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 6000-6010.
[19]	HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[C]// The 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2020: 574.
[20]	WU Y Z, WANG X T, LI Y, et al. Towards vivid and diverse image colorization with generative color prior[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 14357-14366.
[21]	KIM G, KANG K, KIM S, et al. BigColor: colorization using a generative color prior for natural images[C]// The 17th European Conference on Computer Vision. Cham: Springer, 2022: 350-366.
[22]	WANG Y, XIA M H, QI L, et al. PalGAN: image colorization with palette generative adversarial networks[C]// The 17th European Conference on Computer Vision. Cham: Springer, 2022: 271-288.
[23]	HUANG Z T, ZHAO N X, LIAO J. UniColor: a unified framework for multi-modal colorization with transformer[J]. ACM Transactions on Graphics (TOG), 2022, 41(6): 205.
[24]	KANG X Y, YANG T, OUYANG W Q, et al. DDColor: towards photo-realistic image colorization via dual decoders[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 328-338.
[25]	ZHANG R, ZHU J Y, ISOLA P, et al. Real-time user-guided image colorization with learned deep priors[J]. ACM Transactions on Graphics (TOG), 2017, 36(4): 119.
[26]	YUN J, LEE S, PARK M, et al. iColoriT: towards propagating local hints to the right region in interactive colorization by leveraging vision transformer[C]// 2023 IEEE/CVF Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2023: 1787-1796.
[27]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[EB/OL]. [2024-05-01]. https://dblp.org/db/conf/iclr/iclr2021.html#DosovitskiyB0WZ21.
[28]	BAHNG H, YOO S, CHO W, et al. Coloring with words: Guiding image colorization through text-based palette generation[C]// The 15th European Conference on Computer Vision. Cham: Springer, 2018: 443-459.
[29]	WENG S C, WU H, CHANG Z, et al. L-code: language-based colorization using color-object decoupled conditions[C]// The 36th AAAI Conference on Artificial Intelligence. Washington: AAAI, 2022: 2677-2684.
[30]	CHANG Z, WENG S C, LI Y, et al. L-CoDer: language-based colorization with color-object decoupling transformer[C]// The 17th European Conference on Computer Vision. Cham: Springer, 2022: 360-375.
[31]	HE M M, CHEN D D, LIAO J, et al. Deep exemplar-based colorization[J]. ACM Transactions on Graphics (TOG), 2018, 37(4): 47.
[32]	CONG X Y, WU Y, CHEN Q F, et al. Automatic controllable colorization via imagination[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 2609-2619.
[33]	LIANG Z X, LI Z C, ZHOU S C, et al. Control color: multimodal diffusion-based interactive image colorization[J]. [2024-05-07]. https://arxiv.org/abs/2402.10855.
[34]	ZHANG L M, RAO A, AGRAWALA M. Adding conditional control to text-to-image diffusion models[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 3813-3824.
[35]	AKSOY Y, OH T H, PARIS S, et al. Semantic soft segmentation[J]. ACM Transactions on Graphics (TOG), 2018, 37(4): 72.
[36]	KIRILLOV A, MINTUN E, RAVI N, et al. Segment anything[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 3992-4003.