Object depth estimation methods for high photon flux environments

doi:10.11996/JG.j.2095-302X.2025040756

Journal of Graphics ›› 2025, Vol. 46 ›› Issue (4): 756-762.DOI: 10.11996/JG.j.2095-302X.2025040756

• Image Processing and Computer Vision • Previous Articles Next Articles

Object depth estimation methods for high photon flux environments

YANG Jiaxi¹(), YU Letian¹, BAO Qirui¹, BI Sheng², MA Xiaodou¹, Yang Shengqi³, JIANG Yutong⁴, FANG Jianru⁵, WEI Xiaopeng¹, YANG Xin¹()

1. Key Laboratory of Social Computing and Cognitive Intelligence, School of Computer Science, Dalian University of Technology, Dalian Liaoning 116024, China
2. School of Mechanical Engineering, Dalian University of Technology, Dalian Liaoning 116024, China
3. Shenyang Aircraft Design and Research Institute, Aviation Industry Corporation of China, Shenyang Liaoning 110035, China
4. Chinese Scholartree Ridge State Key Laboratory, China North Vehicle Research Institute, Beijing 100072, China
5. Dalian Yaming Automotive Parts Co., Ltd., Dalian Liaoning 116024, China

Received:2024-07-05 Accepted:2025-03-22 Online:2025-08-30 Published:2025-08-11
Contact: YANG Xin
About author:First author contact:
YANG Jiaxi (1999-), master student. His main research interest covers computer vision. E-mail：517542583@qq.com
Supported by:
National Key Research and Development Program of China(2022ZD0210500)

Abstract

Abstract:

The high temporal-resolution and high-precision characteristics of single-photon avalanche diode (SPAD) have opened up a wide range of applications, especially in fields such as computer vision and computational imaging with increasing algorithmic performance demands. Accurate depth estimation can be achieved for various targets using SPAD measurements, however, every time when SPAD device detects a photon, it will enter an undetectable quenching period. When there are a large number of photons in the environment, photon arrivals are more likely to be recorded in earlier bins than later bins, resulting in an obvious histogram distortion towards the shorter temporal axis, while the degree of distortion exacerbates with the increase of photon flux (the number of photons detected per unit time). This phenomenon, known as the Pileup Effect, reduces the accuracy of depth estimation algorithms. In this paper, a SPAD-based prototype was first constructed to collect single-photon measurements under several different photon-flux settings, and a single-photon based dataset was developed to study the pileup effect for depth estimation vision tasks. Based on our dataset, a depth estimation network was then designed to learn photon-flux as global features, simultaneously integrating the local spatial features and global flux-based features from SPAD measurements. Extensive experiments demonstrated that our network significantly achieved superior depth estimation performance under several different photon-flux settings with pileup effects.

Key words: single-photon avalanche diode, photon flux, pileup effect, depth estimation, self-attention mechanism

CLC Number:

TN312.7

YANG Jiaxi, YU Letian, BAO Qirui, BI Sheng, MA Xiaodou, Yang Shengqi, JIANG Yutong, FANG Jianru, WEI Xiaopeng, YANG Xin. Object depth estimation methods for high photon flux environments[J]. Journal of Graphics, 2025, 46(4): 756-762.

Figures/Tables 10

Fig. 1 Network architecture for depth estimation based on SPAD measurements

Fig. 2 Local feature extraction module

Fig. 3 Global feature extraction module with global self-attention guidance block

Fig. 4 Feature fusion and upsample decoder module

Fig. 5 Ten objects and corresponding ground truth map in our dataset

Fig. 6 Experimental results at photon flux of approximately 500 K per second ((a) RGB; (b) GT; (c) Shin; (d) SP3I; (e) PE3I; (f) BPE; (g) Ours)

Fig. 7 Experimental results at photon flux of approximately 1 M per second ((a) RGB; (b) GT; (c) Shin; (d) SP3I; (e) PE3I; (f) BPE; (g) Ours)

Fig. 8 Experimental results at photon flux of approximately 5 M per second ((a) RGB; (b) GT; (c) Shin; (d) SP3I; (e) PE3I; (f) BPE; (g) Ours)

Table 1 Quantitative Results on target depth estimation dataset

光子通量	模型
光子通量	Shin^[14]	SP3I^[7]	PE3I^[9]	BFE^[10]	Ours
每秒约500 K	0.061	0.053	0.037	0.029	0.024
每秒约1 M	0.092	0.098	0.074	0.065	0.033
每秒约5 M	0.141	0.147	0.118	0.110	0.052

Table 2 Ablation analysis on global feature extraction module

光子通量	测试结果
光子通量	不包含全局特征	包含全局特征
每秒约500 K	0.048	0.024
每秒约1 M	0.092	0.033
每秒约5 M	0.141	0.052

References 14

[1]	COVA S, GHIONI M, LOTITO A, et al. Evolution and prospects for single-photon avalanche diodes and quenching circuits[J]. Journal of Modern Optics, 2004, 51(9/10): 1267-1288.
[2]	VELTEN A, WU D, JARABO A, et al. Femto-photography[J]. ACM Transactions on Graphics, 2013, 32(4): 1-8.
[3]	VILLA F, BRONZI D, ZOU Y, et al. CMOS SPADs with up to 500 μm diameter and 55% detection efficiency at 420 nm[J]. Journal of Modern Optics, 2014, 61(2): 102-115.
[4]	GAO L, LIANG J Y, LI C Y, et al. Single-shot compressed ultrafast photography at one hundred billion frames per second[J]. Nature, 2014, 516(7529): 74-77.
[5]	PUSZKA A, DI SIENO L, MORA A D, et al. Spatial resolution in depth for time-resolved diffuse optical tomography using short source-detector separations[J]. Biomedical Optics Express, 2015, 6(1): 1-10. DOI PMID
[6]	HEIDE F, DIAMOND S, LINDELL D B, et al. Sub- picosecond photon-efficient 3D imaging using single-photon sensors[J]. Scientific Reports, 2018, 8(1): 17726.
[7]	LINDELL D B, O'TOOLE M, Wetzstein G. Single-photon 3D imaging with deep sensor fusion[J]. ACM Transactions on Graphics, 2018, 37(4): 1-12.
[8]	TAN H, PENG J Y, XIONG Z W, et al. Deep learning based single-photon 3D imaging with multiple returns[C]// 2020 International Conference on 3D Vision. New York: IEEE Press, 2020: 1196-1205.
[9]	PENG J Y, XIONG Z W, HUANG X, et al. Photon-efficient 3D imaging with a non-local neural network[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 225-241.
[10]	PENG J Y, XIONG Z W, TAN H, et al. Boosting photon- efficient image reconstruction with a unified deep neural network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4): 4180-4197.
[11]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// The 31st International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2017: 6000-6010.
[12]	WANG X L, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7794-7803.
[13]	BAR-DAVID I. Communication under the Poisson regime[J]. IEEE Transactions on Information Theory, 1969, 15(1): 31-37.
[14]	SHIN D, KIRMANI A, GOYAL V K, et al. Photon-efficient computational 3-D and reflectivity imaging with single-photon detectors[J]. IEEE Transactions on Computational Imaging, 2015, 1(2): 112-125.

Object depth estimation methods for high photon flux environments

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 14

Related Articles 13

Recommended Articles

Metrics

Comments

[1]	CHEN Guanhao, XU Dan, HE Kangjian, SHI Hongzhen, ZHANG Hao. TSA-SFNet: transpose self-attention and CNN based stereoscopic fusion network for image super-resolution [J]. Journal of Graphics, 2025, 46(1): 35-46.
[2]	PENG Wen, LIN Jinwei. A short chromosome classification method based on spatial attention and texture enhancement [J]. Journal of Graphics, 2024, 45(5): 1017-1029.
[3]	LIU Zongming, HONG Wei, LONG Rui, ZHU Yue, ZHANG Xiaoyu. Research on automatic generation and application of Ruyuan Yao embroidery based on self-attention mechanism [J]. Journal of Graphics, 2024, 45(5): 1096-1105.
[4]	LI Tao, HU Ting, WU Dandan. Monocular depth estimation combining pyramid structure and attention mechanism [J]. Journal of Graphics, 2024, 45(3): 454-463.
[5]	YU Wei-qun, LIU Jia-tao, ZHANG Ya-ping. Monocular depth estimation based on Laplacian pyramid with attention fusion [J]. Journal of Graphics, 2023, 44(4): 728-738.
[6]	CAO Yi-qin, WU Ming-lin, XU Lu. Steel surface defect detection based on improved YOLOv5 algorithm [J]. Journal of Graphics, 2023, 44(2): 335-345.
[7]	LIAO Zhi-wei , JIN Jing , ZHANG Chao-fan, YANG Xue-zhi. Monocular depth estimation of ASPP networks based on hierarchical compress excitation [J]. Journal of Graphics, 2022, 43(2): 214-222.
[8]	MU Qi , , ZHANG Han , HE Zhi-qiang , LI Zhan-li . Scale adaptive target tracking algorithm based on depth estimation and#br# feature fusion [J]. Journal of Graphics, 2021, 42(4): 563-571.
[9]	JIANG Su-qin , ZHANG Meng-jun , LI Wei-qing , SU Zhi-yong . Real-time virtual and real occlusion processing technology based on voting decision [J]. Journal of Graphics, 2021, 42(4): 629-635.
[10]	CHANG Dong-liang , YIN Jun-hui , XIE Ji-yang , SUN Wei-ya , MA Zhan-yu. Attention-guided Dropout for image classification [J]. Journal of Graphics, 2021, 42(1): 32-36.
[11]	HE Ye, ZHANG Xu-dong, WU Di. FANET: light field depth estimation with multi-channel information fusion [J]. Journal of Graphics, 2020, 41(6): 922-929.
[12]	JIA Rui-ming, LIU Li-qiang, LIU Sheng-jie, CUI Jia-li . Single Image Depth Estimation Based on Encoder-Decoder Convolution Neural Network [J]. Journal of Graphics, 2019, 40(4): 718-724.
[13]	WEN Jing, AN Guo-yan, LIANG Yu-dong . Monocular Image Depth Estimation Based on CNN Features Extraction and Weighted Transfer Learning [J]. Journal of Graphics, 2019, 40(2): 248-255.