基于混合结构的多视图三维场景重建

doi:10.11996/JG.j.2095-302X.2024010199

图学学报 ›› 2024, Vol. 45 ›› Issue (1): 199-208.DOI: 10.11996/JG.j.2095-302X.2024010199

• 计算机图形学与虚拟现实 • 上一篇下一篇

基于混合结构的多视图三维场景重建

周婧怡(), 张栖桐, 冯结青()

浙江大学计算机辅助设计与图形系统全国重点实验室，浙江杭州 310058

收稿日期:2023-07-20 接受日期:2023-12-13 出版日期:2024-02-29 发布日期:2024-02-29
通讯作者:冯结青(1970-)，男，教授，博士。主要研究方向为图形绘制、几何造型、立体视觉、光热太阳能建模与仿真等。E-mail：jqfeng@cad.zju.edu.cn
第一作者:周婧怡(1999-)，女，硕士研究生。主要研究方向为多视图重建。E-mail：22121273@zju.edu.cn
基金资助:
国家自然科学基金项目(61932018);国家自然科学基金项目(62272408)

Hybrid-structure based multi-view 3D scene reconstruction

ZHOU Jingyi(), ZHANG Qitong, FENG Jieqing()

State Key Laboratory of CAD&CG, Zhejiang University, Hangzhou Zhejiang 310058, China

Received:2023-07-20 Accepted:2023-12-13 Published:2024-02-29 Online:2024-02-29
First author：ZHOU Jingyi (1999-), master student. Her main research interest covers multi-view reconstruction. E-mail：22121273@zju.edu.cn
Supported by:
National Natural Science Foundation of China(61932018);National Natural Science Foundation of China(62272408)

摘要/Abstract

摘要：

基于PatchMatch的多视图高效和高精度三维重建一直是一个挑战性问题。红黑棋盘格模式的传播方式并行计算效率高，但对应的视图选择策略精度较差；基于马尔可夫链的视图选择策略能获取更为准确的匹配结果，但算法并行度较低。为了达成场景重建质量与重建时间的平衡，本文提出了一种基于混合结构的多视图三维重建算法，在第一阶段采用沿行/列并行的传播策略和马尔可夫链式的视图选择策略，得到质量较高的初始深度图，并通过引入多层次处理提升弱纹理区域的重建质量；在第二阶段采用棋盘格式传播方式和基于投票的视图选择策略提高计算效率，缩短重建所需时间。通过在Strecha和ETH3D数据集上进行了大量实验和对比表明，本文算法在不降低计算精度的前提下，计算效率提高2.5倍以上。

浙江大学研究生周婧怡、张栖桐和导师冯结青教授提出一种基于混合结构的多视图三维重建算法，在重建算法的效率和重建结果的质量间达成了平衡。算法第一阶段提供质量较高的初始深度图，并通过引入多层次处理提升弱纹理区域的重建质量；第二阶段通过提高算法并行度提升了计算效率，缩短重建所需时间。算法在不降低计算精度的前提下，计算效率提高2.5倍以上。

关键词: 三维重建, 多视图立体匹配, 视图选择, 传播策略, 多层次处理

Abstract:

Achieving accurate and efficient 3D reconstruction through PatchMatch-based multi-view stereo (MVS) algorithms remains a challenging task. The red-black checkerboard propagation method offers high computational efficiency, yet its corresponding view selection strategy lacks accuracy. The view selection strategy based on Markov chain can obtain more accurate matching results, but lacks parallelism. To balance reconstruction quality and runtime, a hybrid-structure based multi-view 3D scene reconstruction algorithm was proposed. In the first stage, the algorithm employed a parallel row/col propagation strategy and a Markov chain-based view selection strategy to produce high-quality initial depth maps. Meanwhile, multi-level processing was utilized to improve the reconstruction quality of weak texture regions. In the second stage, checkerboard propagation and a voting-based view selection strategy were used to increase computational efficiency and reduce reconstruction time. Extensive experiments and comparisons on the Strecha and ETH3D datasets demonstrated that the proposed algorithm can generate results 2.5 times faster without accuracy reduction.

Key words: 3D reconstruction, multi-view stereo matching, view selection, propagation strategy, multi-level processing

中图分类号:

TP391

周婧怡, 张栖桐, 冯结青. 基于混合结构的多视图三维场景重建[J]. 图学学报, 2024, 45(1): 199-208.

ZHOU Jingyi, ZHANG Qitong, FENG Jieqing. Hybrid-structure based multi-view 3D scene reconstruction[J]. Journal of Graphics, 2024, 45(1): 199-208.

图/表 8

参考文献 17

[1]	CAMPBELL N D F, VOGIATZIS G, HERNÁNDEZ C, et al. Using multiple hypotheses to improve depth-maps for multi-view stereo[C]// European Conference on Computer Vision. Berlin, Heidelberg: Springer, 2008: 766-779.
[2]	BARNES C, SHECHTMAN E, FINKELSTEIN A, et al. PatchMatch: a randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 28(3)Article No. 24,
[3]	SHEN S H. Accurate multiple view 3D reconstruction using patch-based stereo for large-scale scenes[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2013, 22(5): 1901-1914. DOI URL
[4]	ZHENG E L, DUNN E, JOJIC V, et al. PatchMatch based joint view selection and depthmap estimation[C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2014: 1510-1517.
[5]	SCHÖNBERGER J L, ZHENG E L, FRAHM J M, et al. Pixelwise view selection for unstructured multi-view stereo[C]// European Conference on Computer Vision. Cham: Springer, 2016: 501-518.
[6]	XU Q S, TAO W B. Multi-scale geometric consistency guided multi-view stereo[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 5478-5487.
[7]	SEITZ S M, CURLESS B, DIEBEL J, et al. A comparison and evaluation of multi-view stereo reconstruction algorithms[C]// 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2006: 519-528.
[8]	FURUKAWA Y, HERNÁNDEZ C. Multi-view stereo: a tutorial[J]. Foundations and Trends® in Computer Graphics and Vision, 2015, 9(1-2): 1-148. DOI URL
[9]	GALLIANI S, LASINGER K, SCHINDLER K. Massively parallel multiview stereopsis by surface normal diffusion[C]// 2015 IEEE International Conference on Computer Vision. New York: IEEE Press, 2016: 873-881.
[10]	YAO Y, LUO Z X, LI S W, et al. MVSNet: depth inference for unstructured multi-view stereo[C]// European Conference on Computer Vision. Cham: Springer, 2018: 785-801.
[11]	GU X D, FAN Z W, ZHU S Y, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 2492-2501.
[12]	WANG F, GALLIANI S, VOGEL C, et al. PatchmatchNet: learned multi-view patchmatch stereo[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press,
[13]	BISHOP C M, NASRABADI N M. Pattern Recognition and Machine Learning[M]// New York: Springer-Verlag Press, 2006: 531-537
[14]	NEAL R M, HINTON G E. A view of the em algorithm that justifies incremental, sparse, and other variants[M]//JORDAN MI. Learning in Graphical Models. Dordrecht: Springer, 1998: 355-368.
[15]	LIAO J E, FU Y P, YAN Q G, et al. Pyramid multi-view stereo with local consistency[J]. Computer Graphics Forum, 2019, 38(7): 335-346. DOI
[16]	SCHÖPS T, SATTLER T, POLLEFEYS M. BAD SLAM: bundle adjusted direct RGB-D SLAM[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 134-144.
[17]	STRECHA C, VON HANSEN W, VAN GOOL L, et al. On benchmarking camera calibration and multi-view stereo for high resolution imagery[C]// 2008 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2008: 1-8.

算法	fountain-P11		HerzJesu-P8
算法	2 cm	10 cm	2 cm	10 cm
COLMAP	0.827	0.975	0.691	0.931
ACMH	0.793	0.952	0.655	0.888
本文算法	0.801	0.956	0.710	0.903

算法	fountain-P11		HerzJesu-P8
算法	2 cm	10 cm	2 cm	10 cm
COLMAP	0.827	0.975	0.691	0.931
ACMH	0.793	0.952	0.655	0.888
本文算法	0.801	0.956	0.710	0.903

参数			均值	court.	deli.	electro	facade	kicker	mea.	office	pipes	play.	relief	relief.	terrace	terrain
5 cm	Acc.	COLMAP	97.09	95.97	97.78	98.56	91.72	97.53	96.01	98.14	98.76	95.28	98.76	98.23	98.77	96.67
		ACMH	96.61	95.29	96.94	98.43	92.05	98.04	95.38	98.07	99.25	93.23	98.35	97.16	97.72	96.00
		本文算法	96.74	96.48	97.72	98.82	93.44	97.64	91.51	97.44	98.59	94.79	98.54	98.02	98.86	95.83
	Comp.	COLMAP	69.91	86.44	82.76	76.59	76.46	63.82	50.92	45.60	49.13	65.87	74.96	74.45	86.30	75.57
		ACMH	72.65	86.19	86.14	75.76	77.63	54.79	46.16	53.28	47.28	77.24	78.91	81.18	90.57	89.33
		本文算法	75.88	87.62	83.49	81.45	79.47	70.37	65.93	50.81	53.81	79.79	78.43	79.42	89.04	86.93
	F₁	COLMAP	80.50	90.96	89.65	86.20	83.40	77.15	66.54	62.27	65.62	77.89	85.23	84.70	92.11	84.83
		ACMH	81.86	90.51	91.22	85.62	84.23	70.29	62.21	69.04	64.05	84.48	87.57	88.45	94.01	92.54
		本文算法	84.50	91.84	90.05	89.30	85.89	81.80	76.65	66.79	69.62	86.64	87.34	87.75	93.69	91.16
10 cm	Acc.	COLMAP	98.75	99.14	98.79	99.30	97.67	98.33	97.82	99.20	99.18	98.76	99.27	98.96	99.29	98.03
		ACMH	98.75	98.86	99.01	99.58	98.10	98.67	97.30	99.18	99.42	98.40	99.19	98.86	98.99	98.19
		本文算法	98.54	99.50	98.88	99.46	98.15	98.40	94.97	98.84	99.02	98.76	99.12	98.89	99.39	97.67
	Comp.	COLMAP	79.47	92.20	90.53	85.30	83.69	78.27	61.47	58.26	62.75	78.28	82.41	82.13	93.83	83.98
		ACMH	79.27	89.52	90.40	80.45	79.77	69.54	54.85	64.83	56.16	83.94	84.58	86.01	95.41	95.03
		本文算法	83.46	91.16	89.64	87.84	85.15	84.02	76.43	62.23	60.66	88.37	85.37	85.28	96.02	92.86
	F₁	COLMAP	87.61	95.54	94.48	91.77	90.14	87.16	75.50	73.41	76.86	87.33	90.05	89.76	96.48	90.46
		ACMH	87.31	93.96	94.51	89.00	87.99	81.58	70.15	78.41	71.78	90.60	91.30	91.99	97.17	96.58
		本文算法	90.01	95.15	94.04	93.29	91.19	90.64	84.70	76.37	75.24	93.27	91.73	91.59	97.68	95.20

参数			均值	court.	deli.	electro	facade	kicker	mea.	office	pipes	play.	relief	relief.	terrace	terrain
5 cm	Acc.	COLMAP	97.09	95.97	97.78	98.56	91.72	97.53	96.01	98.14	98.76	95.28	98.76	98.23	98.77	96.67
		ACMH	96.61	95.29	96.94	98.43	92.05	98.04	95.38	98.07	99.25	93.23	98.35	97.16	97.72	96.00
		本文算法	96.74	96.48	97.72	98.82	93.44	97.64	91.51	97.44	98.59	94.79	98.54	98.02	98.86	95.83
	Comp.	COLMAP	69.91	86.44	82.76	76.59	76.46	63.82	50.92	45.60	49.13	65.87	74.96	74.45	86.30	75.57
		ACMH	72.65	86.19	86.14	75.76	77.63	54.79	46.16	53.28	47.28	77.24	78.91	81.18	90.57	89.33
		本文算法	75.88	87.62	83.49	81.45	79.47	70.37	65.93	50.81	53.81	79.79	78.43	79.42	89.04	86.93
	F₁	COLMAP	80.50	90.96	89.65	86.20	83.40	77.15	66.54	62.27	65.62	77.89	85.23	84.70	92.11	84.83
		ACMH	81.86	90.51	91.22	85.62	84.23	70.29	62.21	69.04	64.05	84.48	87.57	88.45	94.01	92.54
		本文算法	84.50	91.84	90.05	89.30	85.89	81.80	76.65	66.79	69.62	86.64	87.34	87.75	93.69	91.16
10 cm	Acc.	COLMAP	98.75	99.14	98.79	99.30	97.67	98.33	97.82	99.20	99.18	98.76	99.27	98.96	99.29	98.03
		ACMH	98.75	98.86	99.01	99.58	98.10	98.67	97.30	99.18	99.42	98.40	99.19	98.86	98.99	98.19
		本文算法	98.54	99.50	98.88	99.46	98.15	98.40	94.97	98.84	99.02	98.76	99.12	98.89	99.39	97.67
	Comp.	COLMAP	79.47	92.20	90.53	85.30	83.69	78.27	61.47	58.26	62.75	78.28	82.41	82.13	93.83	83.98
		ACMH	79.27	89.52	90.40	80.45	79.77	69.54	54.85	64.83	56.16	83.94	84.58	86.01	95.41	95.03
		本文算法	83.46	91.16	89.64	87.84	85.15	84.02	76.43	62.23	60.66	88.37	85.37	85.28	96.02	92.86
	F₁	COLMAP	87.61	95.54	94.48	91.77	90.14	87.16	75.50	73.41	76.86	87.33	90.05	89.76	96.48	90.46
		ACMH	87.31	93.96	94.51	89.00	87.99	81.58	70.15	78.41	71.78	90.60	91.30	91.99	97.17	96.58
		本文算法	90.01	95.15	94.04	93.29	91.19	90.64	84.70	76.37	75.24	93.27	91.73	91.59	97.68	95.20

算法	court.	deli.	electro	facade	kicker	meadow	office	pipes	play.	relief	relief.	terrace	terrain
COLMAP	71.10	83.47	88.73	153.62	59.00	24.88	50.44	24.44	69.95	65.75	62.80	42.25	88.36
ACMH	17.69	15.96	20.55	40.46	18.59	5.79	9.78	3.64	14.61	17.55	21.02	9.23	24.40
本文算法	28.26	30.46	34.93	61.93	24.51	9.21	19.51	7.74	24.5	25.16	24.55	15.17	36.09

基于混合结构的多视图三维场景重建

Hybrid-structure based multi-view 3D scene reconstruction

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 17

相关文章 15

编辑推荐

Metrics

本文评价

[1]	黄家晖, 穆太江. 动态三维场景重建研究综述[J]. 图学学报, 2024, 45(1): 14-25.
[2]	石敏, 王炳祺, 李兆歆, 朱登明. 一种带高光处理的无缝纹理映射方法[J]. 图学学报, 2024, 45(1): 148-158.
[3]	王江安, 黄乐, 庞大为, 秦林珍, 梁温茜. 基于自适应聚合循环递归的稠密点云重建网络[J]. 图学学报, 2024, 45(1): 230-239.
[4]	成欢, 王硕, 李孟, 秦伦明, 赵芳. 面向自动驾驶场景的神经辐射场综述[J]. 图学学报, 2023, 44(6): 1091-1103.
[5]	薛皓玮, 王美丽. 融合生物力学约束与多模态数据的手部重建[J]. 图学学报, 2023, 44(4): 794-800.
[6]	葛海明, 张维, 王小龙, 朱晶晶, 贾非, 薛亚东. 基于SfM的城市电缆隧道三维重建方法优化研究[J]. 图学学报, 2023, 44(3): 540-550.
[7]	王江安, 庞大为, 黄乐, 秦林珍. 基于多尺度特征递归卷积的稠密点云重建网络 [J]. 图学学报, 2022, 43(5): 875-883.
[8]	白静, 孟庆亮, 徐昊, 范有福, 杨瞻源. ST-Rec3D：基于结构和目标感知的三维重建[J]. 图学学报, 2022, 43(3): 469-477.
[9]	王浩 1，邵堃 1，霍星 2，杨鹏 1，檀结庆 2 . 一种基于极大特征点的三维椎骨分割方法[J]. 图学学报, 2019, 40(1): 40-45.
[10]	邹晓1，陈正鸣1,2，朱红强1，童晶1,2. 基于移动平台的三维虚拟试发型系统实现及应用[J]. 图学学报, 2018, 39(2): 309-316.
[11]	罗昌平1,2，夏海波1，赵琼3. 基于运动视差的三维重建技术在自导引小车中的应用[J]. 图学学报, 2017, 38(4): 615-622.
[12]	缪永伟1,2，冯小红2，林海斌2，张旭东2，高飞2. 2D-3D 转换:由二维线画图生成三维复杂零部件[J]. 图学学报, 2017, 38(2): 162-169.
[13]	陈培兴，王明泉，李世虎，侯慧玲，王玉. 基于形态学和Otsu 的固体火箭发动机CT 缺陷三维分割[J]. 图学学报, 2015, 36(4): 581-586.
[14]	孙正. 血管内超声图像后处理综述[J]. 图学学报, 2015, 36(3): 468-476.
[15]	孙修恩，张典华，孙昕萍. 三维数字重建在青铜器修复中的应用研究[J]. 图学学报, 2014, 35(6): 912-917.