Point cloud feature enhanced 3D object detection in complex indoor scenes

doi:10.11996/JG.j.2095-302X.2025010059

Abstract

Abstract:

3D point cloud object detection in complex indoor scenes presents challenges due to large-scale point clouds and dense objects with many details. When dealing with point cloud data, existing detection algorithms lose a significant amount of local features and fail to extract enough spatial and semantic information, resulting in low detection accuracy. To solve this problem, a point cloud features enhanced 3D object detection in complex indoor scenes (PEF) algorithm was proposed based on an improved VoteNet. Firstly, a dynamic feature compensation module was used to simulate the interactive query process between seed point set features and grouping set features, gradually recovering lost features for feature compensation. Secondly, a residual MLP module was introduced into the feature extraction part, and a deeper feature learning network was constructed through a residual structure to mine more detailed point cloud features. Finally, in the proposal stage, a feature self-attention mechanism was introduced to model the semantic relationship between a set of independent object points, generating a new feature map. Experiments conducted on the public datasets SUN RGB-D and ScanNet V2 demonstrated that the improved model enhanced the detection accuracy for indoor objects by 5.0% and 11.5% respectively on mAP@0.25 compared with the baseline model. Extensive ablation experiments confirmed the effectiveness of each improved module.

Key words: indoor scene, 3D point cloud, object detection, feature compensation, interactive query, residual, self-attention, feature map

CLC Number:

TP391.41

YUAN Chao, ZHAO Mingxue, ZHANG Fengyi, FENG Xiaoyong, LI Bing, CHEN Rui. Point cloud feature enhanced 3D object detection in complex indoor scenes[J]. Journal of Graphics, 2025, 46(1): 59-69.

Figures/Tables 14

References 43

[1]	MCCORMAC J, CLARK R, BLOESCH M, et al. Fusion++: volumetric object-level SLAM[C]// 2018 International Conference on 3D Vision. New York: IEEE Press, 2018: 32-41.
[2]	WANG C, XU D F, ZHU Y K, et al. DenseFusion: 6D object pose estimation by iterative dense fusion[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 3338-3347.
[3]	PARK Y, LEPETIT V, WOO W. Multiple 3D object tracking for augmented reality[C]// 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. New York: IEEE Press, 2008: 117-120.
[4]	张旭, 王智, 崔粲, 等. 二维激光雷达定位建图系统设计与实现[J]. 光学技术, 2019, 45(5): 596-600.
	ZHANG X, WANG Z, CUI C, et al. Design and implementation of 2D lidar positioning and mapping system[J]. Optical Technique, 2019, 45(5): 596-600 (in Chinese).
[5]	ZHOU Y, TUZEL O. VoxeLNet: end-to-end learning for point cloud based 3d object detection[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 4490-4499.
[6]	LANG A H, VORA S, CAESAR H, et al. PointPillars: fast encoders for object detection from point clouds[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 12689-12697.
[7]	HE C H, ZENG H, HUANG J Q, et al. Structure aware single-stage 3D object detection from point cloud[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 11870-11879.
[8]	YE M S, XU S J, CAO T Y. HVNet: hybrid voxel network for lidar based 3d object detection[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1628-1637.
[9]	QI C R, SU H, KAICHUN M, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 77-85.
[10]	QI C R, LI Y, HAO S, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[C]// The 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 5105-5114.
[11]	YANG Z T, SUN Y A, LIU S, et al. 3DSSD: point-based 3D single stage object detector[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 11037-11045.
[12]	SHI S S, WANG X G, LI H S. PointrCNN: 3D object proposal generation and detection from point cloud[C]// Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 770-779.
[13]	QI C R, OR L, HE K M, et al. Deep Hough voting for 3d object detection in point clouds[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 9276-9285.
[14]	周锐闯, 田瑾, 闫丰亭, 等. 融合外部注意力和图卷积的点云分类模型[J]. 图学学报, 2023, 44(6): 1162-1172. DOI
	ZHOU R C, TIAN J, YAN F T, et al. Point cloud classification model incorporating external attention and graph convolution[J]. Journal of Graphics, 2023, 44(6): 1162-1172 (in Chinese).
[15]	XIE Q, LAI Y K, WU J, et al. MLCVNet: multi-level context VoteNet for 3D object detection[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 10444-10453.
[16]	NOH J, LEE S, HAM B. HVPR: hybrid voxel-point representation for single-stage 3D object detection[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 14600-14609.
[17]	NIE J H, HE Z W, YANG Y X, et al. GLT-T: global-local transformer voting for 3D single object tracking in point clouds[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Washington DC: PKP Press, 2023: 1957-1965.
[18]	PAN X R, XIA Z F, SONG S J, et al. 3D object detection with Pointformer[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 7459-7468.
[19]	周静, 胡怡宇, 黄心汉. 形状补全引导的Transformer点云目标检测方法[J]. 智能系统学报, 2023, 18(4): 731-742.
	ZHOU J, HU Y Y, HUANG X H. Shape completion-guided Transformer point cloud object detection method[J]. CAAI Transactions on Intelligent Systems, 2023, 18(4): 731-742 (in Chinese).
[20]	XU M T, DING R Y, ZHAO H S, et al. PAConv: position adaptive convolution with dynamic kernel assembling on point clouds[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 3172-3181.
[21]	LIU Y C, FAN B, XIANG S M, et al. Relation-shape convolutional neural network for point cloud analysis[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 8887-8896.
[22]	THOMAS H, QI C R, DESCHAUD J E, et al. KPConv: flexible and deformable convolution for point clouds[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 6410-6419.
[23]	MA X, QIN C, YOU H, et al. Rethinking network design and local geometry in point cloud: a simple residual MLP framework[EB/OL]. [2024-04-15]. https://arxiv.org/abs/2202.07123?context=cs.
[24]	梁奥, 李峙含, 花海洋. PointMLP-FD: 基于多级自适应下采样的点云分类模型[J]. 图学学报, 2023, 44(1): 112-119. DOI
	LIANG A, LI Z H, HUA H Y. PointMLP-FD: a point cloud classification model based on multi-level adaptive downsampling[J]. Journal of Graphics, 2023, 44(1): 112-119 (in Chinese). DOI
[25]	杨积升, 章云, 李东. 点云目标检测残差投票网络[J]. 广东工业大学学报, 2022, 39(1): 56-62.
	YANG J S, ZHANG Y, LI D. A residual neural network with voting for 3D object detection in point clouds[J]. Journal of Guangdong University of Technology, 2022, 39(1): 56-62 (in Chinese).
[26]	董雯, 林靖宇. 基于语义特征的3D点云室内目标检测[J]. 计算机与数字工程, 2023, 51(6): 1285-1290.
	DONG W, LIN J Y. 3D point cloud indoor object detection based on semantic features[J]. Computer & Digital Engineering, 2023, 51(6): 1285-1290 (in Chinese).
[27]	王婧暄. 面向机器人自动焊接的工件检测与扫描路径规划研究[D]. 哈尔滨: 哈尔滨工业大学, 2023.
	WANG J X. Research on workpiece detection and scan path planning for robot automated welding[D]. Harbin:Harbin Institute of Technology, 2023 (in Chinese).
[28]	李奇. 复杂室内场景三维目标文本描述方法研究[D]. 西安: 西安电子科技大学, 2022.
	LI Q. Research on 3D object caption generation in complex indoor environment[D]. Xi'an: Xidian University, 2022 (in Chinese).
[29]	LIANG Y, FU Y. CascadeV-Det: cascade point voting for 3D object detection[EB/OL]. [2024-07-17]. https://arxiv.org/abs/2401.07477.
[30]	HOU H R, FING M T, WU Z J, et al. 3D object detection from point cloud via voting step diffusion[EB/OL]. [2024-06-02]. https://ieeexplore.ieee.org/document/10602535.
[31]	张新良, 付陈琳, 赵运基. 扩展点态卷积网络的点云分类分割模型[J]. 中国图象图形学报, 2020, 25(8): 1551-1557.
	ZHANG X L, FU C L, ZHAO Y J. Extended pointwise convolution network model for point cloud classification and segmentation[J]. Journal of Image and Graphics, 2020, 25(8): 1551-1557 (in Chinese).
[32]	杨军, 党吉圣. 基于上下文注意力CNN的三维点云语义分割[J]. 通信学报, 2020, 41(7): 195-203. DOI
	YANG J, DANG J S. Semantic segmentation of 3D point cloud based on contextual attention CNN[J]. Journal on Communications, 2020, 41(7): 195-203 (in Chinese). DOI
[33]	DAI A, CHANG A X, SAVVA M, et al. ScanNet: richly-annotated 3D reconstructions of indoor scenes[C]// 2017 IEEE CVFCVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 2432-2443.
[34]	SONG S R, XIAO J X. Deep sliding shapes for amodal 3D object detection in RGB-D images[C]// 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 808-816.
[35]	LAHOUD J, GHANEM B. 2D-driven 3d object detection in RGB-D images[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 4632-4640.
[36]	QI C R, LIU W, WU C, et al. Frustum PointNets for 3D object detection from RGB-D data[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 918-927.
[37]	CHEN J T, LEI B W, SONG Q Y, et al. A hierarchical graph network for 3D object detection on point clouds[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 389-398.
[38]	FENG M T, GILANI S Z, WANG Y N, et al. Relation graph network for 3d object detection in point clouds[J]. IEEE Transactions on Image Processing, 2021, 30: 92-107. DOI PMID
[39]	WANG Y, GUIZILINI V C, ZHANG T, et al. DETR3D: 3D object detection from multi-view images via 3D-to-2D queries[EB/OL]. [2024-06-02]. https://proceedings.mlr.press/v164/wang22b.html.
[40]	CHENG B W, SHENG L, SHI S S, et al. Back-tracing representative points for voting-based 3D object detection in point clouds[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 8959-8968.
[41]	ZHANG Z W, SUN B, YANG H T, et al. H3DNET: 3D object detection using hybrid geometric primitives[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 311-329.
[42]	ZHAO W G, YAN Y Y, YANG C L, et al. Divide and conquer: 3D point cloud instance segmentation with point-wise binarization[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 562-571.
[43]	LIU Z, ZHANG Z, CAO Y, et al. Group-free 3D object detection via transformers[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 2929-2938.

Method	Input	SUN RGB-D		ScanNet V2
Method	Input	mAP @0.25	mAP @0.50	mAP @0.25	mAP @0.50
DSS^[34]	Geo+RGB	42.1	-	15.2	6.8
2D-driven^[35]	Geo+RGB	45.1	-	-	-
F-PointNet^[36]	Geo+RGB	54.0	-	19.8	10.8
HGNet^[37]	Geo only	61.6	-	61.3	34.4
RGNet^[38]	Geo only	59.2	-	48.5	26.0
MLCVNet^[15]	Geo only	59.8	-	64.7	42.1
3DETR^[39]	Geo only	59.1	32.7	65.0	47.0
Pointformer^[18]	Geo only	61.1	-	64.1	42.6
BRNet^[40]	Geo only	61.1	43.7	66.1	50.9
H3DNet^[41]	Geo only	60.1	39.0	67.2	48.1
PBNet^[42]	Geo only	-	-	69.3	60.1
VoteNet^[13]	Geo only	57.7	32.9	58.6	33.5
Ours	Geo only	62.7	39.3	70.1	50.9

Method	Input	SUN RGB-D		ScanNet V2
Method	Input	mAP @0.25	mAP @0.50	mAP @0.25	mAP @0.50
DSS^[34]	Geo+RGB	42.1	-	15.2	6.8
2D-driven^[35]	Geo+RGB	45.1	-	-	-
F-PointNet^[36]	Geo+RGB	54.0	-	19.8	10.8
HGNet^[37]	Geo only	61.6	-	61.3	34.4
RGNet^[38]	Geo only	59.2	-	48.5	26.0
MLCVNet^[15]	Geo only	59.8	-	64.7	42.1
3DETR^[39]	Geo only	59.1	32.7	65.0	47.0
Pointformer^[18]	Geo only	61.1	-	64.1	42.6
BRNet^[40]	Geo only	61.1	43.7	66.1	50.9
H3DNet^[41]	Geo only	60.1	39.0	67.2	48.1
PBNet^[42]	Geo only	-	-	69.3	60.1
VoteNet^[13]	Geo only	57.7	32.9	58.6	33.5
Ours	Geo only	62.7	39.3	70.1	50.9

Method	Cab	Bed	Chair	Sofa	Table	Door	Wind	Bkshf	Pic	Cntr
VoteNet^[13]	38.10	87.92	56.13	89.62	58.77	57.13	37.20	54.70	7.83	88.71
MLCVNet^[15]	42.45	88.84	89.98	87.40	63.50	56.93	46.98	56.94	11.94	63.94
Pointformer^[18]	46.70	88.40	90.50	88.70	65.70	55.00	47.70	55.80	18.00	63.80
H3DNet^[41]	49.40	88.60	91.80	90.20	64.90	61.00	51.90	54.90	18.60	62.00
Group-free^[43]	52.10	91.90	93.60	88.00	70.70	60.70	53.70	62.40	16.10	58.50
Ours	52.44	92.06	94.13	92.11	65.31	63.77	53.76	61.45	33.89	62.10
Method	Desk	Curt	Fridg	Showr	Toil	Sink	Bath	Gabgb	mAP
VoteNet^[13]	71.69	47.23	45.37	47.32	94.94	44.62	92.11	36.27	58.65
MLCVNet^[15]	76.05	56.72	60.86	65.91	98.33	59.18	87.22	47.89	64.48
Pointformer^[18]	69.10	55.40	48.50	66.20	98.90	61.50	86.70	47.40	64.10
H3DNet^[41]	75.90	57.30	57.20	75.30	97.90	67.40	92.50	53.60	67.20
Group-free^[43]	80.90	67.90	47.00	76.30	99.60	72.00	95.30	56.40	69.10
Ours	79.63	56.83	62.06	82.74	99.85	62.78	90.01	56.89	70.10

Method	Cab	Bed	Chair	Sofa	Table	Door	Wind	Bkshf	Pic	Cntr
VoteNet^[13]	38.10	87.92	56.13	89.62	58.77	57.13	37.20	54.70	7.83	88.71
MLCVNet^[15]	42.45	88.84	89.98	87.40	63.50	56.93	46.98	56.94	11.94	63.94
Pointformer^[18]	46.70	88.40	90.50	88.70	65.70	55.00	47.70	55.80	18.00	63.80
H3DNet^[41]	49.40	88.60	91.80	90.20	64.90	61.00	51.90	54.90	18.60	62.00
Group-free^[43]	52.10	91.90	93.60	88.00	70.70	60.70	53.70	62.40	16.10	58.50
Ours	52.44	92.06	94.13	92.11	65.31	63.77	53.76	61.45	33.89	62.10
Method	Desk	Curt	Fridg	Showr	Toil	Sink	Bath	Gabgb	mAP
VoteNet^[13]	71.69	47.23	45.37	47.32	94.94	44.62	92.11	36.27	58.65
MLCVNet^[15]	76.05	56.72	60.86	65.91	98.33	59.18	87.22	47.89	64.48
Pointformer^[18]	69.10	55.40	48.50	66.20	98.90	61.50	86.70	47.40	64.10
H3DNet^[41]	75.90	57.30	57.20	75.30	97.90	67.40	92.50	53.60	67.20
Group-free^[43]	80.90	67.90	47.00	76.30	99.60	72.00	95.30	56.40	69.10
Ours	79.63	56.83	62.06	82.74	99.85	62.78	90.01	56.89	70.10

Method	DFC	RMLP	FSA	SUN RGB-D		ScanNet V2
Method	DFC	RMLP	FSA	mAP @0.25	mAP @0.50	mAP @0.25	mAP @0.50
Baseline	-	-	-	57.7	32.9	58.6	33.5
A	√	-	-	60.1	37.3	66.6	46.9
B	√	√	-	61.3	37.8	68.4	48.0
C	-	-	√	58.2	33.7	62.4	39.9
D	√	√	√	62.7	39.3	70.1	50.9