A seamless texture mapping method with highlight processing

doi:10.11996/JG.j.2095-302X.2024010148

Abstract

Abstract:

As an important step of 3D reconstruction, texture mapping is directly related to the visual effect of the generative model. Classical texture mapping methods usually employ a simple blending method to obtain the surface texture. The corresponding texture regions are obtained by projecting the model surface onto each texture image, and then blending them to obtain the surface texture. However, due to the inaccuracy of the camera pose, the texture mapping results exhibit evident problems such as blurriness and ghosting. In addition, the texture images acquired in the high-light environment tend to contain highlight areas, resulting in texture color loss and diminished texture authenticity. To address these issues, a seamless texture mapping method capable of effectively eliminating highlight reflections was proposed. The method selected the best texture image for each model surface by assessing the quality of the texture image, and chromaticity consistency was utilized to constrain the optimized camera pose, eliminating obvious texture misalignments. To address the highlight problem, a highlight processing module was proposed, utilizing multi-view image information and employing a two-color reflection model to identify and process the highlight texture. Finally, adjustments were made to the texture color consistency to tackle color inconsistency between textures. The experimental results demonstrated that the proposed algorithm can obtain superior texture mapping results compared with state-of-the-art methods and effectively eliminate highlight reflections.

Key words: 3D reconstruction, texture mapping, highlight removal, texture rendering, pose correction

CLC Number:

TP391

SHI Min, WANG Bingqi, LI Zhaoxin, ZHU Dengming. A seamless texture mapping method with highlight processing[J]. Journal of Graphics, 2024, 45(1): 148-158.

Figures/Tables 14

Fig. 1 Overall flow of the algorithm ((a) Input model and calibrated images; (b) Texture selection results; (c) Initial texture mapping results; (d) Camera attitude adjustment; (e) Highlight removal; (f) Uniform colour)

Fig. 2 The result of texture selection ((a) The result of initial texture selection; (b) The right picture is the result of optimization)

Fig. 3 Highlight processing results ((a) The image to be processed; (b) The result of the processing of Ref [11])

Fig. 4 Imaging principle of multi-angle shooting

Fig. 5 Results of highlight texture screening

Fig. 6 Poisson editing example diagram

Fig. 7 The results of real data ((a) Ref [7]; (b) Ref [9]; (c) This paper)

Fig. 8 The results of simulation data ((a) Ref [7]; (b) Ref [9]; (c) This paper; (d) Ground truth)

Table 1 Evaluation results of PSNR and SSIM (A, B represents the corresponding perspective)

模型	PSNR↑			SSIM↑
模型	文献[7]	文献[9]	本文	文献[7]	文献[9]	本文
BowlA	21.235	21.662	25.127	0.822	0.881	0.893
BowlB	18.571	18.742	22.985	0.761	0.829	0.859
BunnyA	25.153	27.431	28.695	0.840	0.922	0.934
BunnyB	17.665	19.088	23.908	0.763	0.853	0.873

Fig. 9 Ablation experiment for color processing ((a) Color consistency adjustment only; (b) Poisson editing only; (c) Combination of two step process)

Fig. 10 The results of texture data without significant highlights ((a) Ref [7]; (b) Ref [9]; (c) This paper)

Fig. 11 Compare with the method of vertex coloring ((a) Ref [1]; (b) Ref [16]; (c) This paper)

Table 2 Time consumption assessment between different methods

模型	场景数据			运行时间/s
模型	顶点数/k	面数/k	纹理图数	文献[7]	文献[9]	本文
Fountain	31.2	59.9	21	8.4	5823.7	31.9
Board	48.1	95.5	6	1.4	1043.1	19.5
Can	82.7	162.5	10	3.7	3525.3	25.2
Floor	25.7	50.0	6	1.9	825.4	21.2
Loin	45.5	58.2	11	3.4	3672.3	17.4
Bowl	76.0	152.1	11	5.8	3824.5	16.8
Bunny	15.3	30.3	14	2.6	3238.7	18.5
Gate	37.3	58.4	16	6.3	5782.4	23.5

Fig. 12 Comparison of texture details between the method in this paper and Fu’s method ((a) This paper; (b) Ref [9])

References 29

[1]	NEWCOMBE R A, IZADI S, HILLIGES O, et al. KinectFusion: real-time dense surface mapping and tracking[C]// 2011 10th IEEE International Symposium on Mixed and Augmented Reality. New York: IEEE Press, 2011: 127-136.
[2]	CAO Y P, KOBBELT L, HU S M. Real-time high-accuracy three-dimensional reconstruction with consumer RGB-D cameras[J]. ACM Transactions on Graphics, 37(5): 171:1-171:16.
[3]	DAI A, NIEßNER M, ZOLLHÖFER M, et al. BundleFusion: real-time globally consistent 3D reconstruction using on-the-fly surface reintegration[J]. ACM Transactions on Graphics, 36(3): 24:1-24:18.
[4]	包永堂, 周鹏飞, 齐越. 面向单幅图像的逼真3D人脸重建方法[J]. 计算机辅助设计与图形学学报, 2022, 34(12): 1850-1858.
	BAO Y T, ZHOU P F, QI Y. Realistic 3D face reconstruction method for single image[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(12): 1850-1858 (in Chinese).
[5]	LEMPITSKY V, IVANOV D. Seamless mosaicing of image-based texture maps[C]// 2007 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2007: 1-6.
[6]	GAL R, WEXLER Y, OFEK E, et al. Seamless montage for texturing models[J]. Computer Graphics Forum, 2010, 29(2): 479-486. DOI URL
[7]	WAECHTER M, MOEHRLE N, GOESELE M. Let there be color! large-scale texturing of 3D reconstructions[M]// Computer Vision - ECCV 2014. Cham: Springer International Publishing, 2014: 836-850.
[8]	FU Y P, YAN Q G, YANG L, et al. Texture mapping for 3D reconstruction with RGB-D sensor[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 4645-4653.
[9]	FU Y P, YAN Q G, LIAO J, et al. Seamless texture optimization for RGB-D reconstruction[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(3): 1845-1859. DOI URL
[10]	GUO J, ZHOU Z J, WANG L M. Single image highlight removal with a sparse and low-rank reflection model[C]// Computer Vision - ECCV 2018. Cham: Springer, 2018: 282-298.
[11]	YAMAMOTO T, NAKAZAWA A. General improvement method of specular component separation using high-emphasis filter and similarity function[J]. ITE Transactions on Media Technology and Applications, 2019, 7(2): 92-102. DOI URL
[12]	BERNARDINI F, MARTIN I M, RUSHMEIER H. High-quality texture reconstruction from multiple scans[J]. IEEE Transactions on Visualization and Computer Graphics, 2001, 7(4): 318-332. DOI URL
[13]	CALLIERI M, CIGNONI P, CORSINI M, et al. Masked photo blending: mapping dense photographic data set on high-resolution sampled 3D models[J]. Computers & Graphics, 2008, 32(4): 464-473. DOI URL
[14]	TOTZ J, CHUNG A J, YANG G Z. Patient-specific texture blending on surfaces of arbitrary topology[EB/OL]. [2023-05-24]. https://www.doc.ic.ac.uk/amiarcs/proceedings2009/AMI-ARCS2009_proceedings.pdf.
[15]	ZHOU Q Y, KOLTUN V. Color map optimization for 3D reconstruction with consumer depth cameras[J]. ACM Transactions on Graphics, 33(4): 155:1-155:10.
[16]	BI S, KALANTARI N K, RAMAMOORTHI R. Patch-based optimization for image-based texture mapping[J]. ACM Transactions on Graphics, 36(4): 106:1-106:11.
[17]	MAIER R, KIM K, CREMERS D, et al. Intrinsic3D: high-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 3133-3141.
[18]	PRADA F, KAZHDAN M, CHUANG M, et al. Gradient- domain processing within a texture atlas[J]. ACM Transactions on Graphics, 37(4): 154:1-154:14.
[19]	KIM J, KIM H, PARK J, et al. Global texture mapping for dynamic objects[J]. Computer Graphics Forum, 2019, 38(7): 697-705. DOI
[20]	COELHO D, DAL'COL L, MADEIRA T, et al. A robust 3D-based color correction approach for texture mapping applications[J]. Sensors, 2022, 22(5): 1730. DOI URL
[21]	DAL'COL L, COELHO D, MADEIRA T, et al. A sequential color correction approach for texture mapping of 3D meshes[J]. Sensors, 2023, 23(2): 607. DOI URL
[22]	LIU X Q, LI J T, LU G D. Generating high-fidelity texture in RGB-D reconstruction using patches density regularization[J]. Computer-Aided Design, 2023, 160: 103516. DOI URL
[23]	HUANG J W, THIES J, DAI A, et al. Adversarial texture optimization from RGB-D scans[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1556-1565.
[24]	MILDENHALL B, SRINIVASAN P P, TANCIK M, et al. NeRF: representing scenes as neural radiance fields for view synthesis[C]// European Conference on Computer Vision. Cham: Springer, 2020: 405-421.
[25]	MÜLLER T, EVANS A, SCHIED C, et al. Instant neural graphics primitives with a multiresolution hash encoding[J]. ACM Transactions on Graphics, 2022, 41(4): 102:1-102:15.
[26]	TAN R T, IKEUCHI K. Separating reflection components of textured surfaces using a single image[M]// Digitally Archiving Cultural Objects. Boston: Springer, 2008: 353-384.
[27]	FU Y P, YAN Q G, LIAO J, et al. Joint texture and geometry optimization for RGB-D reconstruction[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 5949-5958.
[28]	SHAFER S A. Using color to separate reflection components[J]. Color Research & Application, 1985, 10(4): 210-218.
[29]	PÉREZ P, GANGNET M, BLAKE A. Poisson image editing[J]. ACM Transactions on Graphics, 22(3): 313-318. DOI URL