Geometric feature guided multi-level segmentation for object point clouds

doi:10.11996/JG.j.2095-302X.2023040755

Abstract

Abstract:

The proliferation of scanning object point clouds has brought surface shape analysis to the forefront of research in the computer graphics community. Tasks such as structure extraction, shape editing, and human-object interaction necessitate as precise a point cloud part segmentation as possible. However, due to the complexities of shallow geometric feature extraction, shape loss, and noise interference, the part segmentation, especially the small instance part extraction for scanning point clouds, remains relatively difficult. To address this issue, a multi-level part instance segmentation method guided by geometric features was proposed for object point clouds. The concave, convex, and boundary features were extracted based on the local bending extent of the point cloud surface. Firstly, our method segmented the general structure along the most global prominent concave line. Then it subdivided those segments with obvious geometric differences according to the local shallow features. Finally, the method performed concave-convex collaborative segmentation for some segments to obtain the multi-level segmentation results. The three-stage segmentation process from coarse to fine, along the geometric features from deep to shallow, allowed for better consideration of parts with different scales and finer grained part instance segmentation. The experimental results demonstrated that the proposed method could achieve superior segmentation results on both scanning point clouds and sampled point clouds from CAD models. It provided a more precise and effective method for part segmentation of scanning object point clouds.

Key words: point cloud model, 3D surface segmentation, feature detection, convexity-concavity, region growing

CLC Number:

TP391

LIU Yan, XIONG You-yi, HAN Miao-miao, YANG Long. Geometric feature guided multi-level segmentation for object point clouds[J]. Journal of Graphics, 2023, 44(4): 755-763.

Figures/Tables 9

Table 1 Research status of point cloud segmentation

分割程度	室内场景	室内物体(部件)
分割程度	扫描模型	CAD模型	扫描模型
语义级分割	√	√	√
实例级分割	√	√	×

Fig. 1 The holistic flow chart of geometric feature-guided multi-level segmentation

Fig. 2 Example of break points and feature points

Fig. 3 Geometric features of airplane ((a) Global features of airplane; (b) Local features of fuselage)

Fig. 4 The segment results of chair model and toilet model by our method ((a), (e) Original model; (b), (f) The initial segmentation result; (c), (g) The local concave segmentation result; (d), (h) The concave-convex collaborative segmentation result)

Fig. 5 Indoor scanning object point cloud part segmentation results of different methods ((a) LCCP[27]; (b) CPC[28]; (c) PointNet++[35]; (d) Ours; (e) Ground truth) (Random colors for each segment)

Fig. 6 The OBB of every segment of different methods for indoor scanning object point cloud part segmentation ((a) LCCP[27]; (b) CPC[28]; (c) PointNet++[35]; (d) Ours; (e) OBB of ground truth; (f) Model of ground truth)

Fig. 7 Segmentation results of CAD sampling point cloud by our method ((a) Upright piano; (b) Laptop; (c) Grand piano; (d) Man; (e) Woman)

Table 2 Quantitative comparison (recall@boundary indicator) of four methods for part segmentation on different scanning object point clouds

方法	Average	Chair1	Chair2	Chair3	Chair4	Chair5	Sofa1	Sofa2	Sofa3	Sofa4	Sofa5	Toilet1	Toilet2	Table
LCCP^[27]	0.509	0.650	0.663	0.713	0.638	0.630	0.495	0.509	0.392	0.376	0.447	0.296	0.370	0.443
CPC^[28]	0.398	0.456	0.521	0.331	0.558	0.561	0.888	0.149	0.377	0.358	0.244	0.434	0.082	0.212
PointNet++^[35]	0.531	0.260	0.381	0.874	0.561	1.000	0.795	0.437	0.360	0.407	0.645	0.014	0.542	0.626
Ours	0.590	0.559	0.707	0.604	0.639	0.782	0.701	0.600	0.561	0.447	0.805	0.443	0.413	0.412

References 41

[1]	HU R Z, VAN KAICK O, WU B J, et al. Learning how objects function via co-analysis of interactions[J]. ACM Transactions on Graphics, 2016, 35(4): 1-13.
[2]	HU R Z, YAN Z H, ZHANG J W, et al. Predictive and generative neural networks for object functionality[EB/OL]. [2022-06-28]. https://arxiv.org/abs/2006.15520.
[3]	KIM V G, CHAUDHURI S, GUIBAS L, et al. Shape2Pose: human-centric shape analysis[J]. ACM Transactions on Graphics, 2014, 33(4): 1-12.
[4]	JAIN A, THORMÄHLEN T, RITSCHEL T, et al. Exploring shape variations by 3D-model decomposition and part-based recombination[J]. Computer Graphics Forum, 2012, 31(2pt3): 631-640. DOI URL
[5]	OVSJANIKOV M, LI W, GUIBAS L, et al. Exploration of continuous variability in collections of 3D shapes[J]. ACM Transactions on Graphics, 2011, 30(4): 1-10.
[6]	MO K C, ZHU S L, CHANG A X, et al. PartNet: a large-scale benchmark for fine-grained and hierarchical part-level 3D object understanding[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 909-918.
[7]	ZHOU Q Y, PARK J, KOLTUN V. Open3D: a modern library for 3D data processing[EB/OL]. [2022-01-30]. https://arxiv.org/abs/1801.09847.
[8]	DAI A, CHANG A X, SAVVA M, et al. ScanNet: richly-annotated 3D reconstructions of indoor scenes[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 2432-2443.
[9]	YANG L, YAN Q A, FU Y P, et al. Surface reconstruction via fusing sparse-sequence of depth images[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(2): 1190-1203. DOI URL
[10]	LIAO Y B, ZHU H Y, ZHANG Y G, et al. Point cloud instance segmentation with semi-supervised bounding-box mining[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(12): 10159-10170. DOI URL
[11]	HUANG Q G, WANG W Y, NEUMANN U. Recurrent slice networks for 3D segmentation of point clouds[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 2626-2635.
[12]	YE X Q, LI J M, HUANG H X, et al. 3D recurrent neural networks with context fusion for point cloud semantic segmentation[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 415-430.
[13]	CHANG A X, FUNKHOUSER T, GUIBAS L, et al. ShapeNet: an information-rich 3D model repository[EB/OL]. [2022-06-28]. https://arxiv.org/abs/1512.03012.
[14]	WU Z R, SONG S R, KHOSLA A, et al. 3D ShapeNets: a deep representation for volumetric shapes[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 1912-1920.
[15]	XU M T, DING R Y, ZHAO H S, et al. PAConv: position adaptive convolution with dynamic kernel assembling on point clouds[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 3172-3181.
[16]	XU Y F, FAN T Q, XU M Y, et al. SpiderCNN: deep learning on point sets with parameterized convolutional filters[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 90-105.
[17]	WANG Y, SUN Y B, LIU Z W, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 2019, 38(5): 1-12.
[18]	ZHANG K E, HAO M, WANG J, et al. Linked dynamic graph CNN: learning on point cloud via linking hierarchical features[EB/OL]. [2022-06-10]. https://arxiv.org/abs/1904.10014.
[19]	YU F G, LIU K, ZHANG Y, et al. PartNet: a recursive part decomposition network for fine-grained and hierarchical shape segmentation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 9483-9492.
[20]	UY M A, PHAM Q H, HUA B S, et al. Revisiting point cloud classification: a new benchmark dataset and classification model on real-world data[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 1588-1597.
[21]	RUSU R B, COUSINS S. 3D is here: point cloud library (PCL)[C]// 2011 IEEE International Conference on Robotics and Automation. New York: IEEE Press, 2011: 1-4.
[22]	BHANU B, LEE S, HO C C, et al. Range data processing: representation of surfaces by edges[EB/OL]. [2022-06-28]. https://collections.lib.utah.edu/details?id=706994.
[23]	BESL P J, JAIN R C. Segmentation through variable-order surface fitting[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1988, 10(2): 167-192. DOI URL
[24]	CHEN J, CHEN B Q. Architectural modeling from sparsely scanned range data[J]. International Journal of Computer Vision, 2008, 78(2): 223-236. DOI URL
[25]	李仁忠, 刘阳阳, 杨曼, 等. 基于改进的区域生长三维点云分割[J]. 激光与光电子学进展, 2018, 55(5): 325-331.
	LI R Z, LIU Y Y, YANG M, et al. Three-dimensional point cloud segmentation algorithm based on improved region growing[J]. Laser & Optoelectronics Progress, 2018, 55(5): 325-331 (in Chinese).
[26]	汪文琪, 李宗春, 付永健, 等. 基于改进多规则区域生长的点云多要素分割[J]. 光学学报, 2021, 41(5): 198-212.
	WANG W Q, LI Z C, FU Y J, et al. Multi-factor segmentation of point cloud based on improved multi-rule region growing[J]. Acta Optica Sinica, 2021, 41(5): 198-212 (in Chinese).
[27]	STEIN S C, SCHOELER M, PAPON J, et al. Object partitioning using local convexity[C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2014: 304-311.
[28]	SCHOELER M, PAPON J, WÖRGÖTTER F. Constrained planar cuts - Object partitioning for point clouds[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 5207-5215.
[29]	BORRMANN D, ELSEBERG J, LINGEMANN K, et al. The 3D Hough Transform for plane detection in point clouds: a review and a new accumulator design[J]. 3D Research, 2011, 2(2): 3.
[30]	FISCHLER M A, BOLLES R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[J]. Communications of the ACM, 1981, 24(6): 381-395. DOI URL
[31]	HULIK R, SPANEL M, SMRZ P, et al. Continuous plane detection in point-cloud data based on 3D Hough Transform[J]. Journal of Visual Communication and Image Representation, 2014, 25(1): 86-97. DOI URL
[32]	SCHNABEL R, WAHL R, KLEIN R. Efficient RANSAC for point-cloud shape detection[J]. Computer Graphics Forum, 2007, 26(2): 214-226. DOI URL
[33]	田朋举, 花向红, 康停军, 等. 一种基于点云数据的建筑物平面精细分割方法[J]. 测绘科学, 2021, 46(2): 122-129.
	TIAN P J, HUA X H, KANG T J, et al. Fine segmentation of building planar feature from large scale unorganized point cloud[J]. Science of Surveying and Mapping, 2021, 46(2): 122-129 (in Chinese).
[34]	CHARLES R Q, HAO S, MO K C, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 77-85.
[35]	QI C R, YI L, SU H, et al. Pointnet++: deep hierarchical feature learning on point sets in a metric space[C]// The 24th Annual Conference on Neural Information Processing System. Cambridge: MIT Press, 2017: 5205-5114.
[36]	LIU T, YANG Z H, HU S J, et al. Neighbor reweighted local centroid for geometric feature identification[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 29(2): 1545-1558. DOI URL
[37]	NGUYEN K W L, APRILIA A, KHAIRYANTO A, et al. Edge detection from point cloud of worn parts[EB/OL]. [2022-06-28]. https://dr.ntu.edu.sg/handle/10356/88666.
[38]	GUENNEBAUD G, GROSS M. Algebraic point set surfaces[J]. ACM Transactions on Graphics, 2007, 26(3): 23-es. DOI URL
[39]	SHI S S, WANG X G, LI H S. PointRCNN: 3D object proposal generation and detection from point cloud[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 770-779.
[40]	ZHAO L, TAO W B. JSNet: joint instance and semantic segmentation of 3D point clouds[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12951-12958. DOI URL
[41]	吴军, 崔玥, 赵雪梅, 等. SSA-PointNet++: 空间自注意力机制下的3D点云语义分割网络[J]. 计算机辅助设计与图形学学报, 2022, 34(3): 437-448.
	WU J, CUI Y, ZHAO X M, et al. SSA- PointNet ++: a space self-attention CNN for the semantic segmentation of 3D point cloud[J]. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(3): 437-448 (in Chinese).