Multi-view stereo network reconstruction with feature alignment and context-guided

doi:10.11996/JG.j.2095-302X.2024051008

Abstract

Abstract:

To address the problem of poor reconstruction of small features and edge areas in 3D reconstruction, a multi-view 3D reconstruction network based on feature alignment and context-guided was proposed, namely AGA-MVSNet (alignment and context guidance MVSNet). First, a feature alignment module (FA) and a feature selection module (FS) were constructed to combine different levels of the feature pyramid. The features were first aligned and then fused to enhance the feature extraction capabilities of small-sized objects and edge areas. Subsequently, a context guidance module was incorporated into the cost volume regularization to fully utilize surrounding information and solve the problem of poor correlation between cost volumes, thereby improving the accuracy and completeness of three-dimensional reconstruction, with only a slight increase in memory consumption. Finally, experiments were conducted on the DTU dataset. Experimental results demonstrated that the proposed method improved the accuracy by 2.2% and the overall reconstruction quality by 2.5% compared with the benchmark network CasMVSNet. In addition, the performance on the Tanks and Temples dataset was also excellent compared with some known methods, and good point cloud effects were also generated on the BlendedMVS dataset.

Key words: deep learning, multi-view 3D reconstruction, feature alignment, context guidance, 3D attention mechanism

CLC Number:

TP391

XIONG Chao, WANG Yunyan, LUO Yuhao. Multi-view stereo network reconstruction with feature alignment and context-guided[J]. Journal of Graphics, 2024, 45(5): 1008-1016.

Figures/Tables 16

References 25

[1]	LIU Y P, JING T, QU Q, et al. An augmented-reality holographic stereogram based on 3D optical field information manipulation and reconstruction[J]. Frontiers in Physics, 2022, 9: 828825.
[2]	顾珈静, 刘春, 周骁腾, 等. 基于视觉的明清古家具数字文化档案高精度三维重建[J]. 文物保护与考古科学, 2022, 34(2): 22-30.
	GU J J, LIU C, ZHOU X T, et al. A vision-based high-accuracy 3D-reconstruction method for digital cultural archives of Ming and Qing furniture[J]. Sciences of Conservation and Archaeology, 2022, 34(2): 22-30 (in Chinese).
[3]	李露, 孟偲, 武永昌, 等. 无人机对地观测的遥感场景三维重建实验方案[J]. 实验室研究与探索, 2023, 42(3): 235-240.
	LI L, MENG C, WU Y C, et al. Experimental scheme of 3D reconstruction of remote sensing scenes based on ground observation by UAV[J]. Research and Exploration in Laboratory, 2023, 42(3): 235-240 (in Chinese).
[4]	段永飞, 倪建鑫, 严奉奇, 等. 肾三维重建在结节性硬化症相关肾血管平滑肌脂肪瘤治疗中的应用综述[J]. 解放军医学院学报, 2023, 44(2): 173-176.
	DUAN Y F, NI J X, YAN F Q, et al. Research advances in three-dimensional renal reconstruction in treatment of tuberous sclerosis associated renal angiomyolipoma[J]. Academic Journal of Chinese PLA Medical School, 2023, 44(2): 173-176 (in Chinese).
[5]	马建红, 王稀瑶, 陈永霞, 等. 自动驾驶中图像与点云融合方法研究综述[J]. 郑州大学学报: 理学版, 2022, 54(6): 24-33.
	MA J H, WANG X Y, CHEN Y X, et al. A review of research on image and point cloud fusion methods in automatic driving[J]. Journal of Zhengzhou University: Natural Science Edition, 2022, 54(6): 24-33 (in Chinese).
[6]	FU Q C, XU Q S, ONG Y S, et al. Geo-Neus: geometry-consistent neural implicit surfaces learning for multi-view reconstruction[C]// The 36th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2022: 246.
[7]	XIAO Y X, XUE N, WU T F, et al. Level-S²fM: structure from motion on neural level set of implicit surfaces[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 17205-17214.
[8]	TRUONG P, RAKOTOSAONA M J, MANHARDT F, et al. SPARF: neural radiance fields from sparse and noisy poses[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 4190-4200.
[9]	HUANG S S, ZOU Z X, ZHANG Y C, et al. SC-NeuS: consistent neural surface reconstruction from sparse and noisy views[C]// The 38th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2024: 2357-2365.
[10]	YAO Y, LUO Z X, LI S W, et al. MVSNet: depth inference for unstructured multi-view stereo[C]// The 15th European Conference on Computer Vision. Cham: Springer, 2018: 785-801.
[11]	YAO Y, LUO Z X, LI S W, et al. Recurrent MVSNet for high-resolution multi-view stereo depth inference[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 5520-5529.
[12]	GU X D, FAN Z W, ZHU S Y, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 2492-2501.
[13]	ZHANG S, WEI Z W, XU W J, et al. DSC-MVSNet: attention aware cost volume regularization based on depthwise separable convolution for multi-view stereo[J]. Complex & Intelligent Systems, 2023, 9(6): 6953-6969.
[14]	LUO K Y, GUAN T, JU L L, et al. P-MVSNet: learning patch-wise matching confidence aggregation for multi-view stereo[C]// The IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 10451-10460.
[15]	YANG J Y, MAO W, ALVAREZ J M, et al. Cost volume pyramid based depth inference for multi-view stereo[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 4876-4885.
[16]	YANG R H, MIAO W, ZHANG Z X, et al. SA-MVSNet: self-attention-based multi-view stereo network for 3D reconstruction of images with weak texture[J]. Engineering Applications of Artificial Intelligence, 2024, 131: 107800.
[17]	VATS V K, JOSHI S, CRANDALL D J, et al. GC-MVSNet: multi-view, multi-scale, geometrically-consistent multi-view stereo[C]// The IEEE/CVF Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2024: 3230-3240.
[18]	WANG S C, JIANG H, XIANG L. CT-MVSNet: efficient multi-view stereo with cross-scale transformer[C]// The 30th International Conference on Multimedia Modeling. Cham: Springer, 2024: 394-408.
[19]	王江安, 黄乐, 庞大为, 等. 基于自适应聚合循环递归的稠密点云重建网络[J]. 图学学报, 2024, 45(1): 230-239. DOI
	WANG J A, HUANG L, PANG D W, et al. Dense point cloud reconstruction network based on adaptive aggregation recurrent recursion[J]. Journal of Graphics, 2024, 45(1): 230-239 (in Chinese). DOI
[20]	王云艳, 朱镇中, 熊超. 结合注意力机制与路径聚合的多视图三维重建[J]. 重庆理工大学学报: 自然科学, 2023, 37(10): 127-135.
	WANG Y Y, ZHU Z Z, XIONG C. Multi-view 3D reconstruction combining attention mechanism and path aggregation[J]. Journal of Chongqing University of Technology: Natural Science, 2023, 37(10): 127-135 (in Chinese).
[21]	周婧怡, 张栖桐, 冯结青. 基于混合结构的多视图三维场景重建[J]. 图学学报, 2024, 45(1): 199-208. DOI
	ZHOU J Y, ZHANG Q T, FENG J Q. Hybrid-structure based multi-view 3D scene reconstruction[J]. Journal of Graphics, 2024, 45(1): 199-208 (in Chinese). DOI
[22]	岳明齐, 张迎春, 吴立杰, 等. 基于可变形卷积技术的街景图像语义分割算法[J]. 计算机仿真, 2024, 41(3): 219-226, 259.
	YUE M Q, ZHANG Y C, WU L J, et al. A semantic segmentation algorithm for street view images based on deformable convolution technique[J]. Computer Simulation, 2024, 41(3): 219-226, 259 (in Chinese).
[23]	AANS H, JENSEN R R, VOGIATZIS G, et al. Large-scale data for multiple-view stereopsis[J]. International Journal of Computer Vision, 2016, 120(2): 153-168.
[24]	KNAPITSCH A, PARK J, ZHOU Q Y, et al. Tanks and temples: benchmarking large-scale scene reconstruction[J]. ACM Transactions on Graphics, 2017, 36(4): 78.
[25]	YAO Y, LUO Z X, LI S W, et al. BlendedMVS: a large-scale dataset for generalized multi-view stereo networks[C]// The IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1787-1796.

Method	Acc	Comp	Overall
Gipuma	0.283 1	0.873 2	0.578
MVSNet	0.456 2	0.646 2	0.551
R-MVSNet	0.387 4	0.443 5	0.415
P-MVSNet	0.396 2	0.414 5	0.405
CVPMVSNet	0.334 5	0.407 5	0.371
CasMVSNet	0.383 2	0.378 8	0.381
Ours	0.361 2	0.350 8	0.356

Method	Acc	Comp	Overall
Gipuma	0.283 1	0.873 2	0.578
MVSNet	0.456 2	0.646 2	0.551
R-MVSNet	0.387 4	0.443 5	0.415
P-MVSNet	0.396 2	0.414 5	0.405
CVPMVSNet	0.334 5	0.407 5	0.371
CasMVSNet	0.383 2	0.378 8	0.381
Ours	0.361 2	0.350 8	0.356

Method	内存消耗/MB	运行时间/s	整体性误差/mm
CasMVSNet	6 208	0.784	0.381
Ours	7 339	0.927	0.356

Method	内存消耗/MB	运行时间/s	整体性误差/mm
CasMVSNet	6 208	0.784	0.381
Ours	7 339	0.927	0.356

Method	Acc	Comp	Overall
NeuS	1.202 7	0.874 3	1.038
Ours	0.361 2	0.350 8	0.356