Depth completion with large holes based on structure-guided boundary propagation

doi:10.11996/JG.j.2095-302X.2024030548

Abstract

Abstract:

When collecting depth information using consumer-depth cameras, the collected depth information is often influenced by factors such as equipment, environment, and object material, often leading to missing depth information and holes, limiting the application of depth images in subsequent vision tasks. Existing depth-completion algorithms often struggle to effectively address large-area depth missing, resulting in poor complementation effect and poor object boundary maintenance. To tackle these two problems, a depth-completion algorithm for large holes based on structure-guided boundary growth was proposed. First, combined with the boundary information provided by the RGB images, the structure-guided boundary growth strategy was employed to complement the depth loss at the object boundary. Finally, the large holes inside the object were complemented using a combination of large-hole cut-and-fill and mean filtering. The experimental results demonstrated that the algorithm was able to efficiently maintain object boundaries with large missing areas and across missing objects, while being able to complement the depth information of large missing areas. Quantitative and qualitative results on multiple datasets demonstrated the effectiveness of the method.

Key words: segmented image, structural guidance, Bézier curve fitting, large hole completion, boundary propagation

CLC Number:

TP391

ZHAO Sheng, WU Xiaoqun, LIU Xin. Depth completion with large holes based on structure-guided boundary propagation[J]. Journal of Graphics, 2024, 45(3): 548-557.

Figures/Tables 17

References 27

[1]	LE A V, JUNG S W, WON C S. Directional joint bilateral filter for depth images[J]. Sensors, 2014, 14(7): 11362-11378. DOI PMID
[2]	万琴, 朱晓林, 陈国泉, 等. 分层联合双边滤波的深度图修复算法研究[J]. 计算机工程与应用, 2021, 57(6): 184-190. DOI
	WAN Q, ZHU X L, CHEN G Q, et al. Research on depth map restoration algorithm based on hierarchical joint bilateral filter[J]. Computer Engineering and Applications, 2021, 57(6): 184-190 (in Chinese). DOI
[3]	QI F, HAN J Y, WANG P J, et al. Structure guided fusion for depth map inpainting[J]. Pattern Recognition Letters, 2013, 34(1): 70-76.
[4]	WU Y T, LI T M, SHEN I C, et al. Multi-resolution shared representative filtering for real-time depth completion[EB/OL]. [2023-06-07]. https://kevincosner.github.io/publications/Wu2021MSR/paper.pdf.
[5]	PO L M, ZHANG S H, XU X Y, et al. A new multidirectional extrapolation hole-filling method for depth-image-based rendering[C]// 2011 18th IEEE International Conference on Image Processing. New York: IEEE Press, 2011: 2589-2592.
[6]	ATAPOUR A A, BRECKON T. DepthComp: real-time depth image completion based on prior semantic scene segmentation[C]// The British Machine Vision Conference 2017. Guildford: British Machine Vision Association, 2017: 208. 1-208.13.
[7]	GARDUÑO-RAMÓN M A, TEROL-VILLALOBOS I R, OSORNIO-RIOS R A, et al. A new method for inpainting of depth maps from time-of-flight sensors based on a modified closing by reconstruction algorithm[J]. Journal of Visual Communication and Image Representation, 2017, 47: 36-47.
[8]	CHENG X J, WANG P, YANG R G. Learning depth with convolutional spatial propagation network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 42(10): 2361-2379.
[9]	CHENG X J, WANG P, GUAN C Y, et al. CSPN++: learning context and resource aware convolutional spatial propagation networks for depth completion[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 10615-10622.
[10]	LIN Y K, CHENG T, ZHONG Q, et al. Dynamic spatial propagation network for depth completion[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(2): 1638-1646.
[11]	WANG H W, WANG M Y, CHE Z P, et al. RGB-depth fusion GAN for indoor depth completion[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 6209-6218.
[12]	ZHANG Y M, GUO X D, POGGI M, et al. CompletionFormer: depth completion with convolutions and vision transformers[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 18527-18536.
[13]	SUZUKI S, BE K. Topological structural analysis of digitized binary images by border following[J]. Computer Vision, Graphics, and Image Processing, 1985, 30(1): 32-46.
[14]	SCHARSTEIN D, HIRSCHMÜLLER H, KITAJIMA Y, et al. High-resolution stereo datasets with subpixel-accurate ground truth[M]// Lecture Notes in Computer Science. Cham: Springer International Publishing, 2014: 31-42.
[15]	ROBERTS M, RAMAPURAM J, RANJAN A, et al. Hypersim: a photorealistic synthetic dataset for holistic indoor scene understanding[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 10912-10922.
[16]	SILBERMAN N, FERGUS R. Indoor scene segmentation using a structured light sensor[C]// 2011 IEEE International Conference on Computer Vision Workshops. New York: IEEE Press, 2011: 601-608.
[17]	SILBERMAN N, HOIEM D, KOHLI P, et al. Indoor segmentation and support inference from RGBD images[C]// European Conference on Computer Vision. Heidelberg: Springer, 2012: 746-760.
[18]	SONG S R, LICHTENBERG S P, XIAO J X. SUN RGB-D: a RGB-D scene understanding benchmark suite[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 567-576.
[19]	JANOCH A, KARAYEV S, JIA Y Q, et al. A category-level 3-D object dataset: putting the Kinect to work[C]// 2011 IEEE International Conference on Computer Vision Workshops. New York: IEEE Press, 2011: 1168-1174.
[20]	XIAO J X, OWENS A, TORRALBA A. SUN3D: a database of big spaces reconstructed using SfM and object labels[C]// 2013 IEEE International Conference on Computer Vision. New York: IEEE Press, 2013: 1625-1632.
[21]	UHRIG J, SCHNEIDER N, SCHNEIDER L, et al. Sparsity invariant CNNs[C]// 2017 International Conference on 3D Vision. New York: IEEE Press, 2017: 11-20.
[22]	RICHARDT C, STOLL C, DODGSON N A, et al. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos[J]. Computer Graphics Forum, 2012, 31(2pt1): 247-256.
[23]	BARRON J T, POOLE B. The fast bilateral solver[C]// European Conference on Computer Vision. Cham: Springer, 2016: 617-632.
[24]	FU J, LIU J, TIAN H J, et al. Dual attention network for scene segmentation[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 3141-3149.
[25]	LI A, YUAN Z J, LING Y G, et al. A multi-scale guided cascade hourglass network for depth completion[C]// 2020 IEEE Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2020: 32-40.
[26]	SENUSHKIN D, ROMANOV M, BELIKOV I, et al. Decoder modulation for indoor depth completion[C]// 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems. New York: IEEE Press, 2021: 2181-2188.
[27]	VAN GANSBEKE W, NEVEN D, DE BRABANDERE B, et al. Sparse and noisy LiDAR completion with RGB guidance and uncertainty[C]// 2019 16th International Conference on Machine Vision Applications. New York: IEEE Press, 2019: 1-6.

参数	m
h	1	2	3	4
3	0.607 6	0.601 0	0.594 3	0.588 8
4	0.612 1	0.608 0	0.578 1	0.585 5
5	0.610 9	0.602 6	0.581 0	0.598 0

参数	m
h	1	2	3	4
3	0.607 6	0.601 0	0.594 3	0.588 8
4	0.612 1	0.608 0	0.578 1	0.585 5
5	0.610 9	0.602 6	0.581 0	0.598 0

方法	ai_001_001			ai_001_003			ai_001_004			ai_001_006
	16.28%			13.21%			11.29%			12.13%
	SSIM	RMSE	PSNR	SSIM	RMSE	PSNR	SSIM	RMSE	PSNR	SSIM	RMSE	PSNR
DepthComp^[6]	0.992 6	1.134 7	34.214 1	0.997 0	0.920 9	42.095 8	0.993 7	1.237 6	35.842 1	0.995 7	1.241 1	39.453 0
MCBR^[7]	0.982 3	4.002 7	30.032 7	0.983 4	3.302 6	28.429 2	0.978 2	3.299 7	26.074 2	0.982 5	3.322 1	31.888 1
本文算法	0.998 9	0.566 7	53.006 5	0.999 1	0.466 4	52.593 8	0.998 7	1.018 4	46.911 6	0.999 1	0.678 6	51.248 4

方法	ai_001_001			ai_001_003			ai_001_004			ai_001_006
	16.28%			13.21%			11.29%			12.13%
	SSIM	RMSE	PSNR	SSIM	RMSE	PSNR	SSIM	RMSE	PSNR	SSIM	RMSE	PSNR
DepthComp^[6]	0.992 6	1.134 7	34.214 1	0.997 0	0.920 9	42.095 8	0.993 7	1.237 6	35.842 1	0.995 7	1.241 1	39.453 0
MCBR^[7]	0.982 3	4.002 7	30.032 7	0.983 4	3.302 6	28.429 2	0.978 2	3.299 7	26.074 2	0.982 5	3.322 1	31.888 1
本文算法	0.998 9	0.566 7	53.006 5	0.999 1	0.466 4	52.593 8	0.998 7	1.018 4	46.911 6	0.999 1	0.678 6	51.248 4

方法	Buddha		LivingRoom		Outdoor		Table
方法	RMSE	PSNR	RMSE	PSNR	RMSE	PSNR	RMSE	PSNR
M-JBU^[22]	1.50	26.15	1.82	27.97	3.18	25.91	1.85	26.02
FBS^[23]	1.73	26.15	2.16	30.99	2.94	27.09	2.04	29.77
JBF^[1]	2.21	16.17	1.68	23.77	2.90	22.40	1.72	20.76
M-SRF^[4]	1.12	32.17	1.56	33.31	2.44	28.01	1.28	31.71
本文算法	1.11	34.92	1.24	34.45	2.01	32.49	1.17	31.80