Hybrid-structure based multi-view 3D scene reconstruction

doi:10.11996/JG.j.2095-302X.2024010199

Abstract

Abstract:

Achieving accurate and efficient 3D reconstruction through PatchMatch-based multi-view stereo (MVS) algorithms remains a challenging task. The red-black checkerboard propagation method offers high computational efficiency, yet its corresponding view selection strategy lacks accuracy. The view selection strategy based on Markov chain can obtain more accurate matching results, but lacks parallelism. To balance reconstruction quality and runtime, a hybrid-structure based multi-view 3D scene reconstruction algorithm was proposed. In the first stage, the algorithm employed a parallel row/col propagation strategy and a Markov chain-based view selection strategy to produce high-quality initial depth maps. Meanwhile, multi-level processing was utilized to improve the reconstruction quality of weak texture regions. In the second stage, checkerboard propagation and a voting-based view selection strategy were used to increase computational efficiency and reduce reconstruction time. Extensive experiments and comparisons on the Strecha and ETH3D datasets demonstrated that the proposed algorithm can generate results 2.5 times faster without accuracy reduction.

Key words: 3D reconstruction, multi-view stereo matching, view selection, propagation strategy, multi-level processing

CLC Number:

TP391

ZHOU Jingyi, ZHANG Qitong, FENG Jieqing. Hybrid-structure based multi-view 3D scene reconstruction[J]. Journal of Graphics, 2024, 45(1): 199-208.

Figures/Tables 8

References 17

[1]	CAMPBELL N D F, VOGIATZIS G, HERNÁNDEZ C, et al. Using multiple hypotheses to improve depth-maps for multi-view stereo[C]// European Conference on Computer Vision. Berlin, Heidelberg: Springer, 2008: 766-779.
[2]	BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 28(3)Article No. 24,
[3]	SHEN S H. Accurate multiple view 3D reconstruction using patch-based stereo for large-scale scenes[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2013, 22(5): 1901-1914. DOI URL
[4]	ZHENG E L, DUNN E, JOJIC V, et al. PatchMatch based joint view selection and depthmap estimation[C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2014: 1510-1517.
[5]	SCHÖNBERGER J L, ZHENG E L, FRAHM J M, et al. Pixelwise view selection for unstructured multi-view stereo[C]// European Conference on Computer Vision. Cham: Springer, 2016: 501-518.
[6]	XU Q S, TAO W B. Multi-scale geometric consistency guided multi-view stereo[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 5478-5487.
[7]	SEITZ S M, CURLESS B, DIEBEL J, et al. A comparison and evaluation of multi-view stereo reconstruction algorithms[C]// 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2006: 519-528.
[8]	FURUKAWA Y, HERNÁNDEZ C. Multi-view stereo: a tutorial[J]. Foundations and Trends® in Computer Graphics and Vision, 2015, 9(1-2): 1-148. DOI URL
[9]	GALLIANI S, LASINGER K, SCHINDLER K. Massively parallel multiview stereopsis by surface normal diffusion[C]// 2015 IEEE International Conference on Computer Vision. New York: IEEE Press, 2016: 873-881.
[10]	YAO Y, LUO Z X, LI S W, et al. MVSNet: depth inference for unstructured multi-view stereo[C]// European Conference on Computer Vision. Cham: Springer, 2018: 785-801.
[11]	GU X D, FAN Z W, ZHU S Y, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 2492-2501.
[12]	WANG F, GALLIANI S, VOGEL C, et al. PatchmatchNet: learned multi-view patchmatch stereo[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press,
[13]	BISHOP C M, NASRABADI N M. Pattern Recognition and Machine Learning[M]// New York: Springer-Verlag Press, 2006: 531-537
[14]	NEAL R M, HINTON G E. A view of the em algorithm that justifies incremental, sparse, and other variants[M]//JORDAN MI. Learning in Graphical Models. Dordrecht: Springer, 1998: 355-368.
[15]	LIAO J E, FU Y P, YAN Q G, et al. Pyramid multi-view stereo with local consistency[J]. Computer Graphics Forum, 2019, 38(7): 335-346. DOI
[16]	SCHÖPS T, SATTLER T, POLLEFEYS M. BAD SLAM: bundle adjusted direct RGB-D SLAM[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 134-144.
[17]	STRECHA C, VON HANSEN W, VAN GOOL L, et al. On benchmarking camera calibration and multi-view stereo for high resolution imagery[C]// 2008 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2008: 1-8.

算法	fountain-P11		HerzJesu-P8
算法	2 cm	10 cm	2 cm	10 cm
COLMAP	0.827	0.975	0.691	0.931
ACMH	0.793	0.952	0.655	0.888
本文算法	0.801	0.956	0.710	0.903

算法	fountain-P11		HerzJesu-P8
算法	2 cm	10 cm	2 cm	10 cm
COLMAP	0.827	0.975	0.691	0.931
ACMH	0.793	0.952	0.655	0.888
本文算法	0.801	0.956	0.710	0.903

参数			均值	court.	deli.	electro	facade	kicker	mea.	office	pipes	play.	relief	relief.	terrace	terrain
5 cm	Acc.	COLMAP	97.09	95.97	97.78	98.56	91.72	97.53	96.01	98.14	98.76	95.28	98.76	98.23	98.77	96.67
		ACMH	96.61	95.29	96.94	98.43	92.05	98.04	95.38	98.07	99.25	93.23	98.35	97.16	97.72	96.00
		本文算法	96.74	96.48	97.72	98.82	93.44	97.64	91.51	97.44	98.59	94.79	98.54	98.02	98.86	95.83
	Comp.	COLMAP	69.91	86.44	82.76	76.59	76.46	63.82	50.92	45.60	49.13	65.87	74.96	74.45	86.30	75.57
		ACMH	72.65	86.19	86.14	75.76	77.63	54.79	46.16	53.28	47.28	77.24	78.91	81.18	90.57	89.33
		本文算法	75.88	87.62	83.49	81.45	79.47	70.37	65.93	50.81	53.81	79.79	78.43	79.42	89.04	86.93
	F₁	COLMAP	80.50	90.96	89.65	86.20	83.40	77.15	66.54	62.27	65.62	77.89	85.23	84.70	92.11	84.83
		ACMH	81.86	90.51	91.22	85.62	84.23	70.29	62.21	69.04	64.05	84.48	87.57	88.45	94.01	92.54
		本文算法	84.50	91.84	90.05	89.30	85.89	81.80	76.65	66.79	69.62	86.64	87.34	87.75	93.69	91.16
10 cm	Acc.	COLMAP	98.75	99.14	98.79	99.30	97.67	98.33	97.82	99.20	99.18	98.76	99.27	98.96	99.29	98.03
		ACMH	98.75	98.86	99.01	99.58	98.10	98.67	97.30	99.18	99.42	98.40	99.19	98.86	98.99	98.19
		本文算法	98.54	99.50	98.88	99.46	98.15	98.40	94.97	98.84	99.02	98.76	99.12	98.89	99.39	97.67
	Comp.	COLMAP	79.47	92.20	90.53	85.30	83.69	78.27	61.47	58.26	62.75	78.28	82.41	82.13	93.83	83.98
		ACMH	79.27	89.52	90.40	80.45	79.77	69.54	54.85	64.83	56.16	83.94	84.58	86.01	95.41	95.03
		本文算法	83.46	91.16	89.64	87.84	85.15	84.02	76.43	62.23	60.66	88.37	85.37	85.28	96.02	92.86
	F₁	COLMAP	87.61	95.54	94.48	91.77	90.14	87.16	75.50	73.41	76.86	87.33	90.05	89.76	96.48	90.46
		ACMH	87.31	93.96	94.51	89.00	87.99	81.58	70.15	78.41	71.78	90.60	91.30	91.99	97.17	96.58
		本文算法	90.01	95.15	94.04	93.29	91.19	90.64	84.70	76.37	75.24	93.27	91.73	91.59	97.68	95.20

参数			均值	court.	deli.	electro	facade	kicker	mea.	office	pipes	play.	relief	relief.	terrace	terrain
5 cm	Acc.	COLMAP	97.09	95.97	97.78	98.56	91.72	97.53	96.01	98.14	98.76	95.28	98.76	98.23	98.77	96.67
		ACMH	96.61	95.29	96.94	98.43	92.05	98.04	95.38	98.07	99.25	93.23	98.35	97.16	97.72	96.00
		本文算法	96.74	96.48	97.72	98.82	93.44	97.64	91.51	97.44	98.59	94.79	98.54	98.02	98.86	95.83
	Comp.	COLMAP	69.91	86.44	82.76	76.59	76.46	63.82	50.92	45.60	49.13	65.87	74.96	74.45	86.30	75.57
		ACMH	72.65	86.19	86.14	75.76	77.63	54.79	46.16	53.28	47.28	77.24	78.91	81.18	90.57	89.33
		本文算法	75.88	87.62	83.49	81.45	79.47	70.37	65.93	50.81	53.81	79.79	78.43	79.42	89.04	86.93
	F₁	COLMAP	80.50	90.96	89.65	86.20	83.40	77.15	66.54	62.27	65.62	77.89	85.23	84.70	92.11	84.83
		ACMH	81.86	90.51	91.22	85.62	84.23	70.29	62.21	69.04	64.05	84.48	87.57	88.45	94.01	92.54
		本文算法	84.50	91.84	90.05	89.30	85.89	81.80	76.65	66.79	69.62	86.64	87.34	87.75	93.69	91.16
10 cm	Acc.	COLMAP	98.75	99.14	98.79	99.30	97.67	98.33	97.82	99.20	99.18	98.76	99.27	98.96	99.29	98.03
		ACMH	98.75	98.86	99.01	99.58	98.10	98.67	97.30	99.18	99.42	98.40	99.19	98.86	98.99	98.19
		本文算法	98.54	99.50	98.88	99.46	98.15	98.40	94.97	98.84	99.02	98.76	99.12	98.89	99.39	97.67
	Comp.	COLMAP	79.47	92.20	90.53	85.30	83.69	78.27	61.47	58.26	62.75	78.28	82.41	82.13	93.83	83.98
		ACMH	79.27	89.52	90.40	80.45	79.77	69.54	54.85	64.83	56.16	83.94	84.58	86.01	95.41	95.03
		本文算法	83.46	91.16	89.64	87.84	85.15	84.02	76.43	62.23	60.66	88.37	85.37	85.28	96.02	92.86
	F₁	COLMAP	87.61	95.54	94.48	91.77	90.14	87.16	75.50	73.41	76.86	87.33	90.05	89.76	96.48	90.46
		ACMH	87.31	93.96	94.51	89.00	87.99	81.58	70.15	78.41	71.78	90.60	91.30	91.99	97.17	96.58
		本文算法	90.01	95.15	94.04	93.29	91.19	90.64	84.70	76.37	75.24	93.27	91.73	91.59	97.68	95.20

算法	court.	deli.	electro	facade	kicker	meadow	office	pipes	play.	relief	relief.	terrace	terrain
COLMAP	71.10	83.47	88.73	153.62	59.00	24.88	50.44	24.44	69.95	65.75	62.80	42.25	88.36
ACMH	17.69	15.96	20.55	40.46	18.59	5.79	9.78	3.64	14.61	17.55	21.02	9.23	24.40
本文算法	28.26	30.46	34.93	61.93	24.51	9.21	19.51	7.74	24.5	25.16	24.55	15.17	36.09