Latent structure prior-prompted diffusion model for scene text image super-resolution

doi:10.11996/JG.j.2095-302X.2026051015

Abstract

Abstract:

Scene Text Image Super-Resolution (STISR) aims to recover clear, complete, and recognizable character structures from Low-Resolution (LR) text images, thereby providing high-quality visual inputs for scene text recognition, intelligent document processing, and visual retrieval in natural environments. Owing to factors such as imaging distance, focal variation, motion blur, and complex background interference, LR scene text images often contain broken strokes, blurred character boundaries, distorted contours, and missing structural details, reducing the accuracy and robustness of downstream text recognition models. Existing STISR methods mainly focus on visual texture reconstruction or text-related semantic guidance and remain inadequate for recovering character contours and fine-grained structures under severe degradation. Although diffusion models show strong detail generation capability, without explicit structural constraints, the denoising process may introduce local artifacts, structural shifts, or incomplete stroke recovery when processing highly degraded text images. To address these problems, a Latent Structure Prior-Prompted Diffusion Model (LSPDiff) was constructed for scene text image super-resolution. LSPDiff comprised a Latent Structure Generation Model (LSGM) and a Prior-Prompted Diffusion Model (PPDM). The LSGM took bicubic-interpolated low-resolution text images as input, learned multi-scale latent structural features through an encoder-decoder architecture, and used multiscale supervision to enhance character strokes, edge contours, local textures, and spatial arrangements at different resolutions. Multi-scale supervision is introduced to encourage the generated structural representations to approach the corresponding high-resolution text images. During reverse denoising, the PPDM jointly incorporated the low-resolution conditional image, noisy image, and structural prompts, and a prior-prompt attention module was designed to model the interaction between diffusion intermediate features and latent structural priors, enabling structural information to guide noise estimation and character reconstruction. Since the two models had different optimization objectives, LSGM focuses on structural prior learning while the PPDM focuses on diffusion-based noise prediction, their optimization objectives are not fully consistent. a two-stage training strategy was therefore adopted. The LSGM was first trained independently to obtain stable structural priors, after which its parameters were frozen, and the PPDM was optimized with fixed structural guidance to improve convergence stability and prior transfer effectiveness. Experiments were conducted on the TextZoom benchmark dataset, which contains 21 740 low-resolution and high-resolution scene text image pairs, including 17 367 pairs for training and 4 373 pairs for testing. The test set is divided into easy, medium, and hard subsets according to image degradation difficulty, allowing a detailed evaluation under different restoration conditions. Three scene text recognition models, namely ASTER, MORAN, and CRNN, were used to evaluate recognition accuracy on the restored images. Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) were also adopted to measure reconstruction quality. LSPDiff achieved average recognition accuracies of 66.2%, 63.3%, and 57.0% under ASTER, MORAN, and CRNN, respectively, as well as a PSNR of 22.32 dB and an SSIM of 0.792 2, demonstrating favorable performance in both recognition-oriented restoration and image reconstruction quality. Compared with representative STISR methods, LSPDiff obtains competitive overall results, especially in recognition accuracy and PSNR. Ablation experiments further verify the role of latent structural guidance. Compared with the PPDM-only variant, showed that introducing the LSGM increased PSNR from 21.40 dB to 22.32 dB and recognition accuracy from 60.5% to 66.2%. These results indicate that multi-scale latent structural priors provide effective guidance for recovering character strokes and maintaining text layout consistency. Multi-scale latent structural priors can effectively constrain the diffusion-based reconstruction process and improve stroke continuity, contour clarity, and text readability in low-resolution scene text images, and provide an effective approach to STISR under complex degradation conditions and has potential application value in scene text recognition, intelligent document processing, and visual information understanding.

Key words: scene text image super-resolution, latent structure prior, diffusion model, prior prompt, text recognition

CLC Number:

TP 391.41

LI Kexin, FAN Wanshu, MI Jiale, HAN Yuanpeng, FU Luyan, ZHANG Shixiang. Latent structure prior-prompted diffusion model for scene text image super-resolution[J]. Journal of Graphics, 2026, 47(5): 1015-1024.

Figures/Tables 12

References 37

[1]	DONG C LOY, C C, HE K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. DOI PMID
[2]	LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5835-5843.
[3]	ZHANG Y L, TIAN Y P, KONG Y, et al. Residual dense network for image super-resolution[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 2472-2481.
[4]	MA J Q, LIANG Z T, ZHANG L. A text attention network for spatial deformation robust scene text image super- resolution[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 5911-5920.
[5]	WANG W J, XIE E Z, LIU X B, et al. Scene text image super-resolution in the wild[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 650-666.
[6]	ZHAO M Y, WANG M, BAI F, et al. C3-STISR:scene text image super-resolution with triple clues[EB/OL]. [2025-11-08]. https://dblp.uni-trier.de/db/conf/ijcai/ijcai2022.html#conf/ijcai/ZhaoW0LWZ22.
[7]	GUO H, DAI T, MENG G H, et al. Towards robust scene text image super-resolution via explicit location enhancement[EB/OL]. [2025-11-08]. https://dl.acm.org/doi/abs/10.24963/ijcai.2023/87.
[8]	FENG B T, SMITH J, RUBINSTEIN M, et al. Score-based diffusion models as principled priors for inverse imaging[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 10486-10497.
[9]	XIA B, ZHANG Y L, WANG S Y, et al. DiffIR: efficient diffusion model for image restoration[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 13095-13105.
[10]	HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[C]// The 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates, Inc., 2020: 574.
[11]	HO J, SAHARIA C, CHAN W, et al. Cascaded diffusion models for high fidelity image generation[J]. The Journal of Machine Learning Research, 2022, 23(1): 47.
[12]	CHENG Z Z, XU Y L, BAI F, et al. AON: towards arbitrarily-oriented text recognition[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 5571-5579.
[13]	SHI B G, YANG M K, WANG X G, et al. ASTER: an attentional scene text recognizer with flexible rectification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(9): 2035-2048. DOI PMID
[14]	SHI B G, BAI X, YAO C. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(11): 2298-2304. DOI PMID
[15]	LUO C J, JIN L W, SUN Z H. MORAN: a multi-object rectified attention network for scene text recognition[J]. Pattern Recognition, 2019, 90: 109-118. DOI URL
[16]	ZHAO C R, FENG S Y, ZHAO B N, et al. Scene text image super-resolution via parallelly contextual attention network[C]// The 29th ACM International Conference on Multimedia. New York: ACM, 2021: 2908-2917.
[17]	MA J Q, GUO S, ZHANG L. Text prior guided scene text image super-resolution[J]. IEEE Transactions on Image Processing, 2023, 32: 1341-1353. DOI URL
[18]	ROMBACH R, BLATTMANN A, LORENZ D, et al. High-resolution image synthesis with latent diffusion models[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 10684-10695.
[19]	SOHL-DICKSTEIN J, WEISS E A, MAHESWARANATHAN N, et al. Deep unsupervised learning using nonequilibrium thermodynamics[EB/OL]. [2025-11-08]. https://dblp.uni-trier.de/db/conf/icml/icml2015.html#conf/icml/Sohl-DicksteinW15.
[20]	LI H Y, YANG Y F, CHANG M, et al. SRDiff: single image super-resolution with diffusion probabilistic models[J]. Neurocomputing, 2022, 479: 47-59. DOI URL
[21]	SAHARIA C, CHAN W, CHANG H W, et al. Palette: image-to-image diffusion models[C]// ACM SIGGRAPH 2022 Conference Proceedings. New York: ACM, 2022: 15.
[22]	WANG J Y, YUE Z S, ZHOU S C, et al. Exploiting diffusion prior for real-world image super-resolution[EB/OL]. [2025-11-08]. http://arxiv.org/abs/2305.07015.
[23]	RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]// Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015: 234-241.
[24]	HUANG Z L, WANG X G, HUANG L C, et al. CCNet: criss-cross attention for semantic segmentation[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 603-612.
[25]	HUYNH-THU Q, GHANBARI M. Scope of validity of PSNR in image/video quality assessment[J]. Electronics Letters, 2008, 44(13): 800-801. DOI URL
[26]	WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612. DOI PMID
[27]	ZHU X Y, GUO K H, FANG H, et al. Gradient-based graph attention for scene text image super-resolution[C]// The 37th AAAI Conference on Artificial Intelligence. Washington: AAAI Press, 2023: 3861-3869.
[28]	KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL]. [2025-11-08]. https://dblp.uni-trier.de/db/conf/iclr/iclr2015.html#conf/iclr/KingmaB14.
[29]	LIU B L, YANG Z Y, WANG P F, et al. TextDiff: mask-guided residual diffusion models for scene text image super-resolution[EB/OL]. [2025-11-08]. https://arxiv.org/abs/2308.06743.
[30]	LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 4681-4690.
[31]	ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]// The 15th European Conference on Computer Vision. Cham: Springer, 2018: 286-301.
[32]	CHEN J Y, LI B, XUE X Y. Scene text telescope: text-focused scene image super-resolution[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 12026-12035.
[33]	DAI T, CAI J, ZHANG Y B, et al. Second-Order Attention Network for Single Image Super-Resolution[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 11065-11074.
[34]	ZHU S P, ZHAO Z Y, FANG P F, et al. Improving scene text image super-resolution via dual prior modulation network[C]// The AAAI Conference on Artificial Intelligence, Washington: AAAI, 2023: 3843-3851.
[35]	YU L, LI X J, YU Q, et al. Scene text image super-resolution via textual reasoning and multiscale cross-convolution[J]. Applied Intelligence, 2024, 54(2): 1997-2008. DOI
[36]	李大海, 吕春桂, 王振东. 基于双分支序列残差注意力的场景文本图像超分辨率重建[J]. 计算机工程, 2024, 50(9): 286-295. DOI
	LI D H, LÜ C G, WANG Z D. Scene text image super-resolution reconstruction based on dual-branched sequence residual attention[J]. Computer Engineering, 2024, 50(9): 286-295 (in Chinese). DOI
[37]	YE K R, GARCIA DORADO I, RAPTIS M, et al. TextSR: diffusion super-resolution with multilingual OCR guidance[EB/OL]. [2025-11-08]. http://arxiv.org/abs/2505.23119.

Method	PSNR	SSIM	LPIPS
TSRN^[5]	21.42	0.769 1	0.097 3
TATT^[4]	21.52	0.793 0	0.094 5
C3-STISR^[6]	21.60	0.793 1	0.094 4
TextDiff^[29]	21.37	0.784 1	0.082 2
LSPDiff (Ours)	22.32	0.792 2	0.089 3

Method	PSNR	SSIM	LPIPS
TSRN^[5]	21.42	0.769 1	0.097 3
TATT^[4]	21.52	0.793 0	0.094 5
C3-STISR^[6]	21.60	0.793 1	0.094 4
TextDiff^[29]	21.37	0.784 1	0.082 2
LSPDiff (Ours)	22.32	0.792 2	0.089 3

Method	Accuracy of ASTER				Accuracy of MORAN				Accuracy of CRNN
Method	easy	medium	hard	average	easy	medium	hard	average	easy	medium	hard	average
Bicubic	64.7	42.4	31.2	47.2	60.6	37.9	30.8	44.1	36.4	21.1	21.1	26.8
SRCNN^[1]	69.4	43.4	32.2	49.5	63.2	39.0	30.2	45.3	38.7	21.6	20.9	27.7
SRResNet^[30]	69.4	47.3	34.3	51.3	60.7	42.9	32.6	46.3	39.7	27.6	22.7	30.6
RCAN^[31]	67.3	46.6	35.1	50.7	63.1	42.9	33.6	47.5	46.8	27.9	26.5	34.5
SAN^[33]	68.1	48.7	36.2	50.7	65.6	44.4	35.2	49.4	50.1	31.2	28.1	37.2
TSRN^[5]	75.1	56.3	40.1	58.3	70.1	53.3	37.9	54.8	52.5	38.2	31.4	41.4
TBSRN^[32]	75.7	59.9	41.6	60.0	74.1	57.0	40.8	58.4	59.6	47.1	35.3	48.1
PCAN^[16]	77.5	60.7	43.1	61.5	73.7	57.6	41.0	58.5	59.6	45.4	34.8	47.4
TATT^[4]	78.9	63.4	45.4	63.6	72.5	60.2	43.1	59.5	62.6	53.4	39.8	52.6
C3-STISR^[6]	79.1	63.3	46.8	64.1	74.2	61.0	43.2	60.5	65.2	53.6	39.8	53.7
DPMN (+TATT)^[34]	79.3	64.1	45.2	63.9	73.3	61.5	43.9	60.4	64.4	54.2	39.2	53.4
TSAN^[27]	79.6	64.1	45.3	64.1	78.4	61.3	45.1	62.7	64.6	53.3	38.8	53.0
LEMMA^[7]	81.1	66.3	47.4	66.0	77.7	64.4	44.6	63.2	67.1	58.8	40.6	56.3
TRMCC^[35]	79.1	63.1	47.1	64.1	73.7	60.0	43.7	60.1	63.4	53.1	38.9	52.6
DSRASRN^[36]	80.9	64.6	46.2	65.0	75.4	62.6	45.6	62.1	61.6	53.6	38.6	52.0
TextSR^[37]	80.8	64.1	48.3	65.4	77.7	61.9	46.2	63.0	67.8	57.8	41.8	56.6
LSPDiff (Ours)	81.2	67.0	47.4	66.2	78.6	63.0	45.4	63.3	67.6	58.5	42.7	57.0
HR	94.2	87.7	76.2	86.6	91.2	85.3	74.2	84.1	76.4	75.1	64.6	72.4

Method	Accuracy of ASTER				Accuracy of MORAN				Accuracy of CRNN
Method	easy	medium	hard	average	easy	medium	hard	average	easy	medium	hard	average
Bicubic	64.7	42.4	31.2	47.2	60.6	37.9	30.8	44.1	36.4	21.1	21.1	26.8
SRCNN^[1]	69.4	43.4	32.2	49.5	63.2	39.0	30.2	45.3	38.7	21.6	20.9	27.7
SRResNet^[30]	69.4	47.3	34.3	51.3	60.7	42.9	32.6	46.3	39.7	27.6	22.7	30.6
RCAN^[31]	67.3	46.6	35.1	50.7	63.1	42.9	33.6	47.5	46.8	27.9	26.5	34.5
SAN^[33]	68.1	48.7	36.2	50.7	65.6	44.4	35.2	49.4	50.1	31.2	28.1	37.2
TSRN^[5]	75.1	56.3	40.1	58.3	70.1	53.3	37.9	54.8	52.5	38.2	31.4	41.4
TBSRN^[32]	75.7	59.9	41.6	60.0	74.1	57.0	40.8	58.4	59.6	47.1	35.3	48.1
PCAN^[16]	77.5	60.7	43.1	61.5	73.7	57.6	41.0	58.5	59.6	45.4	34.8	47.4
TATT^[4]	78.9	63.4	45.4	63.6	72.5	60.2	43.1	59.5	62.6	53.4	39.8	52.6
C3-STISR^[6]	79.1	63.3	46.8	64.1	74.2	61.0	43.2	60.5	65.2	53.6	39.8	53.7
DPMN (+TATT)^[34]	79.3	64.1	45.2	63.9	73.3	61.5	43.9	60.4	64.4	54.2	39.2	53.4
TSAN^[27]	79.6	64.1	45.3	64.1	78.4	61.3	45.1	62.7	64.6	53.3	38.8	53.0
LEMMA^[7]	81.1	66.3	47.4	66.0	77.7	64.4	44.6	63.2	67.1	58.8	40.6	56.3
TRMCC^[35]	79.1	63.1	47.1	64.1	73.7	60.0	43.7	60.1	63.4	53.1	38.9	52.6
DSRASRN^[36]	80.9	64.6	46.2	65.0	75.4	62.6	45.6	62.1	61.6	53.6	38.6	52.0
TextSR^[37]	80.8	64.1	48.3	65.4	77.7	61.9	46.2	63.0	67.8	57.8	41.8	56.6
LSPDiff (Ours)	81.2	67.0	47.4	66.2	78.6	63.0	45.4	63.3	67.6	58.5	42.7	57.0
HR	94.2	87.7	76.2	86.6	91.2	85.3	74.2	84.1	76.4	75.1	64.6	72.4

Method	PSNR	SSIM	Accuracy/%
TSRN^[5]	21.42	0.769 1	51.5
TATT^[4]	21.52	0.793 0	58.6
C3-STISR^[6]	21.60	0.793 1	59.4
DPMN (+TATT)^[34]	21.49	0.792 5	59.2
LEMMA^[7]	22.09	0.754 9	62.3
TSAN^[27]	22.10	0.783 5	64.1
TRMCC^[35]	21.58	0.792 0	64.1
DSRASRN^[36]	21.69	0.7806	65.0
LSPDiff (Ours)	22.32	0.7922	66.2