3D shape completion via deep learning: a method survey

doi:10.11996/JG.j.2095-302X.2023020201

Abstract

Abstract:

The task of 3D shape completion, a fundamental aspect of computer graphics and computer vision, has been widely employed in many fields. 3D shape completion aims to infer complete shapes from partially missing shape data. This paper reviewed the current 3D model completion algorithms based on deep learning, and analyzed their advantages and disadvantages. According to the different forms of descriptors, the 3D model completion algorithms could be broadly classified into two categories: the completion methods based on 2D shape descriptors and the completion methods based on 3D shape descriptors. The former involved the projecting of the 3D model into the 2D space for feature extraction to obtain a complete model, including 3D model completion methods based on 2D images and depth maps. The latter involved the direct use of 3D representation for model completion, and according to different representations of 3D models, could be further divided into voxel-based, point cloud-based and implicit-based methods. Meanwhile, this survey provided an overview of the commonly used datasets, metrics, and state-of-the-art performance in the field. It also analyzed and discussed the problems facing current 3D model completion algorithms based on deep learning, and suggested potential avenues for future research.

Key words: 3D shape completion, deep learning, convolutional neural network, computer vision, multi-layer perceptron

CLC Number:

TP391.41

YANG Liu, WU Xiao-qun. 3D shape completion via deep learning: a method survey[J]. Journal of Graphics, 2023, 44(2): 201-215.

Figures/Tables 21

References 65

[1]	ENGEL J, SCHÖPS T, CREMERS D. LSD-SLAM: large-scale direct monocular SLAM[M]//Computer Vision - ECCV 2014. Cham: Springer International Publishing, 2014: 834-849.
[2]	HOU J, DAI A, NIEßNER M. 3D-SIS: 3D semantic instance segmentation of RGB-D scans[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 4416-4425.
[3]	BOUD A C, HANIFF D J, BABER C, et al. Virtual reality and augmented reality as a training tool for assembly tasks[C]// 1999 IEEE International Conference on Information Visualization. New York: IEEE Press, 1999: 32-36.
[4]	CHANG A X, FUNKHOUSER T, GUIBAS L, et al. ShapeNet: an information-rich 3D model repository[EB/OL]. [2021-12-09]. https://arxiv.org/abs/1512.03012.
[5]	YU X M, RAO Y M, WANG Z Y, et al. PoinTr: diverse point cloud completion with geometry-aware transformers[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 12478-12487.
[6]	YUAN W T, KHOT T, HELD D, et al. PCN: point completion network[C]// 2018 International Conference on 3D Vision. New York: IEEE Press, 2018: 728-737.
[7]	TCHAPMI L P, KOSARAJU V, REZATOFIGHI H, et al. TopNet: structural point cloud decoder[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 383-392.
[8]	HUANG Z T, YU Y K, XU J W, et al. PF-net: point fractal network for 3D point cloud completion[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 7659-7667.
[9]	ZHANG X C, FENG Y T, LI S Q, et al. View-guided point cloud completion[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 15885-15894.
[10]	PAN L, CHEN X Y, CAI Z A, et al. Variational relational point completion network[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 8520-8529.
[11]	WU Z R, SONG S R, KHOSLA A, et al. 3D ShapeNets: a deep representation for volumetric shapes[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 1912-1920.
[12]	GEIGER A, LENZ P, STILLER C, et al. Vision meets robotics: the KITTI dataset[J]. The International Journal of Robotics Research, 2013, 32(11): 1231-1237. DOI URL
[13]	GU J Y, MA W C, MANIVASAGAM S, et al. Weakly- supervised 3D shape completion in the wild[M]// Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 283-299.
[14]	DAI A, CHANG A X, SAVVA M, et al. ScanNet: richly- annotated 3D reconstructions of indoor scenes[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 2432-2443.
[15]	ZHANG S L, LI S, HAO A M, et al. Point clou semantic scene completion from RGB-D images[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(4): 3385-3393. DOI URL
[16]	HU T, HAN Z Z, SHRIVASTAVA A, et al. Render4Completion: synthesizing multi-view depth maps for 3D shape completion[C]// 2019 IEEE/CVF International Conference on Computer Vision Workshop. New York: IEEE Press, 2019: 4114-4122.
[17]	HU T, HAN Z Z, ZWICKER M. 3D shape completion with multi-view consistent inference[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 10997-11004. DOI URL
[18]	CHARLES R Q, HAO S, MO K C, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 77-85.
[19]	QI C R, YI L, SU H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[C]// The 31st International Conference on Neural Information Processing Systems. New York: ACM, 2017: 5105-5114.
[20]	DAI A, QI C R, NIEßNER M. Shape completion using 3D-encoder-predictor CNNs and shape synthesis[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 6545-6554.
[21]	HAN X G, LI Z, HUANG H B, et al. High-resolution shape completion using deep neural networks for global structure and local geometry inference[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 85-93.
[22]	GKIOXARI G, JOHNSON J, MALIK J. Mesh R-CNN[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 9784-9794.
[23]	WANG P S, LIU Y, TONG X. Deep octree-based CNNs with output-guided skip connections for 3D shape and scene completion[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2020: 1074-1081.
[24]	XIE H Z, YAO H X, ZHOU S C, et al. GRNet: gridding residual network for dense point cloud completion[M]// Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 365-381.
[25]	GUO Q, WANG Z J, JUEFEI-XU F, et al. CarveNet: carving point-block for complex 3D shape completion[EB/OL]. [2022-04-21]. https://arxiv.org/abs/2107.13452.
[26]	WANG X G, ANG M H, LEE G H. Voxel-based network for shape completion by leveraging edge generation[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 13169-13178.
[27]	WEN X, XIANG P, HAN Z Z, et al. PMP-net: point cloud completion by learning multi-step point moving paths[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 7439-7448.
[28]	WEN X, LI T Y, HAN Z Z, et al. Point cloud completion by skip-attention network with hierarchical folding[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1936-1945.
[29]	WANG X G, ANG M H, LEE G H. Cascaded refinement network for point cloud completion[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 787-796.
[30]	WANG X G, ANG M H, LEE G H. Point cloud completion by learning shape priors[C]// 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems. New York: IEEE Press, 2020: 10719-10726.
[31]	WANG Y D, TAN D J, NAVAB N, et al. SoftPoolNet: shape descriptor for point cloud completion and classification[M]// Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 70-85.
[32]	ZONG D M, SUN S L, ZHAO J. ASHF-net: adaptive sampling and hierarchical folding network for robust point cloud completion[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(4): 3625-3632. DOI URL
[33]	GROUEIX T, FISHER M, KIM V G, et al. A papier-Mache approach to learning 3D surface generation[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 216-224.
[34]	LIU M H, SHENG L, YANG S, et al. Morphing and sampling network for dense point cloud completion[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 11596-11603. DOI URL
[35]	SON H, KIM Y M. SAUM: symmetry-aware upsampling module for consistent point cloud completion[C]// Computer Vision - ACCV 2020: 15th Asian Conference on Computer Vision, Revised Selected Papers, Part I. New York: ACM, 2020: 158-174.
[36]	ZHANG W X, YAN Q G, XIAO C X. Detail preserved point cloud completion via separated feature aggregation[M]// Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 512-528.
[37]	NIE Y Y, LIN Y Q, HAN X G, et al. Skeleton-bridged point completion: from global inference to local adjustment[EB/OL]. [2022-04-21]. https://arxiv.org/abs/2010.07428.
[38]	WANG Y, SUN Y B, LIU Z W, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 2019, 38(5): 146.
[39]	PAN L. ECG: edge-aware point cloud completion with graph convolution[J]. IEEE Robotics and Automation Letters, 2020, 5(3): 4392-4398. DOI URL
[40]	WANG K Q, CHEN K, JIA K. Deep cascade generation on point sets[C]// The 28th International Joint Conference on Artificial Intelligence. New York: ACM, 2019: 3726-3732.
[41]	SHI J Q, XU L Y, HENG L, et al. Graph-guided deformation for point cloud completion[J]. IEEE Robotics and Automation Letters, 2021, 6(4): 7081-7088. DOI URL
[42]	ZHU L P, WANG B Y, TIAN G Y, et al. Towards point cloud completion: point rank sampling and cross-cascade graph CNN[J]. Neurocomputing, 2021, 461: 1-16. DOI URL
[43]	ALLIEGRO A, VALSESIA D, FRACASTORO G, et al. Denoise and contrast for category agnostic shape completion[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 4627-4636.
[44]	GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144. DOI URL
[45]	SARMAD M, LEE H J, KIM Y M. RL-GAN-net: a reinforcement learning agent controlled GAN network for real-time point cloud shape completion[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 5891-5900.
[46]	XIE C L, WANG C X, ZHANG B, et al. Style-based point generator with adversarial rendering for point cloud completion[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 4617-4626.
[47]	CHEN X L, CHEN B Q, MITRA N J. Unpaired point cloud completion on real scans using adversarial training[EB/OL]. [2022-04-21]. https://arxiv.org/abs/1904.00069.
[48]	ZHANG J Z, CHEN X Y, CAI Z A, et al. Unsupervised 3D shape completion through GAN inversion[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 1768-1777.
[49]	WEN X, HAN Z Z, CAO Y P, et al. Cycle4Completion: unpaired point cloud completion using cycle transformation with missing region coding[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 13075-13084.
[50]	CAI Y J, LIN K Y, ZHANG C, et al. Learning a structured latent space for unsupervised point cloud completion[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 5533-5543.
[51]	XIANG P, WEN X, LIU Y S, et al. SnowflakeNet: point cloud completion by snowflake point deconvolution with skip-transformer[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 5479-5489.
[52]	YU L Q, LI X Z, FU C W, et al. PU-net: point cloud upsampling network[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 2790-2799.
[53]	YU L Q, LI X Z, FU C W, et al. EC-net: an edge-aware point set consolidation network[M]//Computer Vision - ECCV 2018. Cham: Springer International Publishing, 2018: 398-414.
[54]	QIAN Y, HOU J H, KWONG S, et al. PUGeo-net: a geometry-centric network for 3D point cloud upsampling[M]// Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 752-769.
[55]	WANG Y F, WU S H, HUANG H, et al. Patch-based progressive 3D point set upsampling[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 5951-5960.
[56]	QIAN G C, ABUALSHOUR A, LI G H, et al. PU-GCN: point cloud upsampling using graph convolutional networks[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 11678-11687.
[57]	LUO L Q, TANG L L, ZHOU W Y, et al. PU-EVA: an edge-vector based approximation solution for flexible-scale point cloud upsampling[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 16188-16197.
[58]	LI R H, LI X Z, FU C W, et al. PU-GAN: a point cloud upsampling adversarial network[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 7202-7211.
[59]	PARK J J, FLORENCE P, STRAUB J, et al. DeepSDF: learning continuous signed distance functions for shape representation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 165-174.
[60]	CHIBANE J, ALLDIECK T, PONS-MOLL G. Implicit functions in feature space for 3D shape reconstruction and completion[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 6968-6979.
[61]	PING Y H, WEI G D, YANG L, et al. Self-attention implicit function networks for 3D dental data completion[J]. Computer Aided Geometric Design, 2021, 90: 1-12.
[62]	YAN W, ZHANG R N, WANG J, et al. Vaccine-style-net: point cloud completion in implicit continuous function space[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 2067-2075.
[63]	YAN X G, LIN L Q, MITRA N J, et al. ShapeFormer: transformer-based shape completion via sparse representation[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 6229-6239.
[64]	TRETSCHK E, TEWARI A, GOLYANIK V, et al. PatchNets: patch-based generalizable deep implicit 3D shape representations[C]// Computer Vision - ECCV 2020: 16th European Conference. New York: ACM, 2020: 293-309.
[65]	CHABRA R, LENSSEN J E, ILG E, et al. Deep local shapes: learning local SDF priors for detailed 3D reconstruction[M]// Computer Vision - ECCV 2020. Cham: Springer International Publishing, 2020: 608-625.

名称	出处	相机数	类别数	模型总数
PCN dataset^[6]	PCN	8	8	30 974
Completion3D^[7]	TOP-Net	1	8	29 774
PF-Net dataset^[8]	PF-Net	-	13	14 473
ShapeNet-ViPC^[9]	ViPC	24	13	38 328
MVP dataset^[10]	VRC-Net	26	16	140 000

名称	出处	相机数	类别数	模型总数
PCN dataset^[6]	PCN	8	8	30 974
Completion3D^[7]	TOP-Net	1	8	29 774
PF-Net dataset^[8]	PF-Net	-	13	14 473
ShapeNet-ViPC^[9]	ViPC	24	13	38 328
MVP dataset^[10]	VRC-Net	26	16	140 000

方法	年份	思想	是否依赖GroundTruth完整模型	缺点	数据集
MVCN^[16]	2019	多视图补全	是	需要完整模型训练	ShapeNet KITTI
MVCI^[17]	2020	一致性推理	是	补全精度不佳	ShapeNet KITTI
Weakly-supervised^[13]	2020	弱监督补全	否	遗漏细节	ShapeNet
ViPC^[18]	2021	视图获得信息	是	-	ShapeNet-VIPC

方法	年份	思想	是否依赖GroundTruth完整模型	缺点	数据集
MVCN^[16]	2019	多视图补全	是	需要完整模型训练	ShapeNet KITTI
MVCI^[17]	2020	一致性推理	是	补全精度不佳	ShapeNet KITTI
Weakly-supervised^[13]	2020	弱监督补全	否	遗漏细节	ShapeNet
ViPC^[18]	2021	视图获得信息	是	-	ShapeNet-VIPC

方法	年份	思想	优点	缺点	数据集
3D-EPN^[20]	2017	体素补全	捕获形状信息	低分辨率输出	ShapeNet
3D-FCN^[21]	2017	局部几何细化	提炼局部几何	- 数据指数级增长 -	ShapeNet
Mesh R-CNN^[22]	2019	利用网格顶点边缘卷积进行细化	提高边缘感知		ShapeNet
O-CNNs^[23]	2020	体素转化为八叉树	提高补全鲁棒性		ShapeNet
GR-Net^[24]	2020	单元格加权卷积	空间感知特征	传感器影响	PCN datasets TopNet datasets KITTI
CarveNet^[25]	2021	逐点卷积	减少冗余点	时间空间复杂度高	ShapeNet KITTI
VE-PCN^[26]	2021	体素结合边缘信息	重建高分辨率	-	PCN datasets TopNet datasets