Physically-augmented and depth synergized freehand 3D ultrasound reconstruction

doi:10.11996/JG.j.2095-302X.2026040726

Abstract

Abstract:

Ultrasound, as a radiation-free and portable imaging modality, extends traditional 2D ultrasound utility through 3D reconstruction by integrating 2D cross-sectional slices to provide richer spatial contextual information. Existing sensor-free 3D freehand reconstruction either relies on cumbersome external localization or on deep-learning surrogates that overlook long-range spatial dependencies and poorly-textured regions. Therefore, PD²BNet, a physically-augmented, depth-synergized framework was proposed, which organically achieved complementary fusion of “internal physical interpretability and external deep generalization.” At the input stage, a three-branch parallel feature-decoupled encoder was designed to independently model grayscale texture, Canny geometric edges, and optical-flow motion vectors, thereby circumventing semantic conflicts among heterogeneous physical priors in low-level convolutions. During spatiotemporal fusion, a CNN-ConvLSTM architecture was embedded to recursively associate local topology with temporal evolution, suppressing instantaneous pose jumps. A composite physics-regularized loss was constructed, jointly incorporating velocity smoothing and physical prior regularization to significantly mitigate inter-frame jitter and accumulated drift, achieving robustness in low-texture ultrasound pullback scenarios. By combining explicit physical constraints with deep representations, accurate, sensor-free 6-DoF pose estimation was achieved, overcoming the interpretability and generalization limits of either paradigm alone. On Freehand_US_data and TUS-REC-Challenge, PD²BNet reduced final drift rate to 16.97 % and 18.24 %, respectively, with mean angular errors of merely 3.01° and 3.27°, outperforming state-of-the-art methods and achieving millimeter-level localization accuracy under typical clinical scanning protocols such as forearm examination. Quantitative and qualitative analyses confirmed that the synergy between physics augmentation and deep modeling drove the performance gains, establishing PD²BNet as a high-precision, robust algorithmic framework for clinical sensorless freehand 3D ultrasound reconstruction.

Key words: freehand ultrasound localization, 3D ultrasound reconstruction, multi-feature physical priors, physics-deep dual drive, deep learning

CLC Number:

TP391

LI Yuhua, JIANG Shan, YANG Zhiyong, WANG Yuze, ZHOU Zeyang. Physically-augmented and depth synergized freehand 3D ultrasound reconstruction[J]. Journal of Graphics, 2026, 47(4): 726-735.

Figures/Tables 12

References 22

[1]	ZHOU R, GUO F M, AZARPAZHOOH M R, et al. A voxel-based fully convolution network and continuous max-flow for carotid vessel-wall-volume segmentation from 3D ultrasound images[J]. IEEE Transactions on Medical Imaging, 2020, 39(9): 2844-2855. DOI PMID
[2]	YANG X, DOU H R, HUANG R B, et al. Agent with warm start and adaptive dynamic termination for plane localization in 3D ultrasound[J]. IEEE Transactions on Medical Imaging, 2021, 40(7): 1950-1961. DOI URL
[3]	LI Y H, JIANG S, YANG Z Y, et al. Deformable brain pMRI-iUS registration based on higher-order cliques Markov random field[J]. Biomedical Signal Processing and Control, 2026, 113: 109146. DOI URL
[4]	PREVOST R, SALEHI M, JAGODA S, et al. 3D freehand ultrasound without external tracking using deep learning[J]. Medical Image Analysis, 2018, 48: 187-202. DOI PMID
[5]	MOZAFFARI M H, LEE W S. Freehand 3-D ultrasound imaging: a systematic review[J]. Ultrasound in Medicine & Biology, 2017, 43(10): 2099-2124. DOI URL
[6]	PU G, JIANG S, YANG Z Y, et al. A novel ultrasound probe calibration method for multimodal image guidance of needle placement in cervical cancer brachytherapy[J]. Physica Medica, 2022, 100: 81-89. DOI URL
[7]	GAO H T, HUANG Q H, XU X M, et al. Wireless and sensorless 3D ultrasound imaging[J]. Neurocomputing, 2016, 195: 159-171. DOI URL
[8]	MIURA K, ITO K, AOKI T, et al. Pose estimation of 2D ultrasound probe from ultrasound image sequences using CNN and RNN[C]// The 2nd International Workshop on Advances in Simplifying Medical Ultrasound. Cham: Springer, 2021: 96-105.
[9]	MERCIER L, LANG T, LINDSETH F, et al. A review of calibration techniques for freehand 3-D ultrasound systems[J]. Ultrasound in Medicine & Biology, 2005, 31(2): 143-165. DOI URL
[10]	SUN R, LIU C B, WANG W S, et al. UltrasOM: a mamba-based network for 3D freehand ultrasound reconstruction using optical flow[J]. Computer Methods and Programs in Biomedicine, 2025, 268: 108843. DOI URL
[11]	PREVOST R, SALEHI M, SPRUNG J, et al. Deep learning for sensorless 3D freehand ultrasound imaging[C]// The 20th International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2017: 628-636.
[12]	GUO H T, CHAO H Q, XU S, et al. Ultrasound volume reconstruction from freehand scans without tracking[J]. IEEE Transactions on Biomedical Engineering, 2023, 70(3): 970-979. DOI URL
[13]	MOHAMED F, SIANG C V. A survey on 3D ultrasound reconstruction techniques[M]//ACEVES-FERNANDEZ M A. Artificial Intelligence-Applications in Medicine and Biology. London: IntechOpen, 2019: 73-92.
[14]	FARNEBÄCK G. Two-frame motion estimation based on polynomial expansion[C]// The 13th Scandinavian Conference on Image Analysis. Cham: Springer, 2003: 363-370.
[15]	CIPOLLA R, GAL Y, KENDALL A. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7482-7491.
[16]	LI Q, SHEN Z Y, LI Q, et al. Long-term dependency for 3D reconstruction of freehand ultrasound without external tracker[J]. IEEE Transactions on Biomedical Engineering, 2024, 71(3): 1033-1042. DOI URL
[17]	LI Q, SHEN Z Y, YANG Q Y, et al. Nonrigid reconstruction of freehand ultrasound without a tracker[C]// The 27th International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2024: 689-699.
[18]	LI Q, SHEN Z Y, LI Q, et al. Trackerless freehand ultrasound with sequence modelling and auxiliary transformation over past and future frames[C]// The 20th International Symposium on Biomedical Imaging. New York: IEEE Press, 2023: 1-5.
[19]	LUO M Y, YANG X, WANG H Z, et al. RecON: Online learning for sensorless freehand 3D ultrasound reconstruction[J]. Medical Image Analysis, 2023, 87: 102810. DOI URL
[20]	NING G C, LIANG H Y, ZHOU L, et al. Spatial position estimation method for 3D ultrasound reconstruction based on hybrid transfomers[C]// The 19th International Symposium on Biomedical Imaging. New York: IEEE Press, 2022: 1-5.
[21]	YAN Z N, YANG X, LUO M Y, et al. Fine-grained context and multi-modal alignment for freehand 3D ultrasound reconstruction[C]// The 27th International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2024: 340-349.
[22]	DOU Y M, MU F Z, LI Y, et al. Sensorless end-to-end freehand 3-D ultrasound reconstruction with physics-guided deep learning[J]. IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control, 2024, 71(11): 1514-1525. DOI URL

编码器分支	输入特征(维度)	层级	操作	输出通道	特征图尺寸
分支A：灰度流	原始灰度图	1	Conv2D(1→32, k=3, p=1) + BN + ReLU + MaxPool2D(2)	32	H/2×W/2
		2	Conv2D(32→64, k=3, p=1) + BN + ReLU + MaxPool2D(2)	64	H/4×W/4
		3	Conv2D(64→128, k=3, p=1) + BN + ReLU + MaxPool2D(2)	128	H/8×W/8
分支B：边缘流	Canny距离变换	1	Conv2D(1→32, k=3, p=1) + BN + ReLU + MaxPool2D(2)	32	H/2×W/2
		2	Conv2D(32→64, k=3, p=1) + BN + ReLU + MaxPool2D(2)	64	H/4×W/4
		3	Conv2D(64→128, k=3, p=1) + BN + ReLU + MaxPool2D(2)	128	H/8×W/8
分支C：运动流	双向光流张量	1	Conv2D(4→32, k=3, p=1) + BN + ReLU + MaxPool2D(2)	32	H/2×W/2
		2	Conv2D(32→64, k=3, p=1) + BN + ReLU + MaxPool2D(2)	64	H/4×W/4
		3	Conv2D(64→128, k=3, p=1) + BN + ReLU + MaxPool2D(2)	128	H/8×W/8

编码器分支	输入特征(维度)	层级	操作	输出通道	特征图尺寸
分支A：灰度流	原始灰度图	1	Conv2D(1→32, k=3, p=1) + BN + ReLU + MaxPool2D(2)	32	H/2×W/2
		2	Conv2D(32→64, k=3, p=1) + BN + ReLU + MaxPool2D(2)	64	H/4×W/4
		3	Conv2D(64→128, k=3, p=1) + BN + ReLU + MaxPool2D(2)	128	H/8×W/8
分支B：边缘流	Canny距离变换	1	Conv2D(1→32, k=3, p=1) + BN + ReLU + MaxPool2D(2)	32	H/2×W/2
		2	Conv2D(32→64, k=3, p=1) + BN + ReLU + MaxPool2D(2)	64	H/4×W/4
		3	Conv2D(64→128, k=3, p=1) + BN + ReLU + MaxPool2D(2)	128	H/8×W/8
分支C：运动流	双向光流张量	1	Conv2D(4→32, k=3, p=1) + BN + ReLU + MaxPool2D(2)	32	H/2×W/2
		2	Conv2D(32→64, k=3, p=1) + BN + ReLU + MaxPool2D(2)	64	H/4×W/4
		3	Conv2D(64→128, k=3, p=1) + BN + ReLU + MaxPool2D(2)	128	H/8×W/8

模型编号	CNN层数	物理增强通道	物理先验正则损失	Speed loss	${{\varepsilon }_{frame}}$↓	${{\varepsilon }_{acc}}$↓	${{\varepsilon }_{drift}}$↓	FDR/%↓	MEA(°)↓
1	1	√	√	√	0.68±0.21	32.90±10.18	56.76±21.08	41.62±14.62	7.52±3.86
2	3	×	×	√	0.64±0.21	31.32±10.60	52.72±17.65	38.41±12.47	8.15±5.44
3	3	√	√	×	0.52±0.19	25.52±9.13	47.20±17.93	35.12±12.31	5.13±2.85
4	3	√	×	√	0.31±0.20	16.11±9.94	29.33±18.63	21.17±13.10	3.32±1.40
5	3	√	√	√	0.27±0.06	13.13±10.51	23.75±10.61	16.97±4.38	3.01±0.96

模型编号	CNN层数	物理增强通道	物理先验正则损失	Speed loss	${{\varepsilon }_{frame}}$↓	${{\varepsilon }_{acc}}$↓	${{\varepsilon }_{drift}}$↓	FDR/%↓	MEA(°)↓
1	1	√	√	√	0.68±0.21	32.90±10.18	56.76±21.08	41.62±14.62	7.52±3.86
2	3	×	×	√	0.64±0.21	31.32±10.60	52.72±17.65	38.41±12.47	8.15±5.44
3	3	√	√	×	0.52±0.19	25.52±9.13	47.20±17.93	35.12±12.31	5.13±2.85
4	3	√	×	√	0.31±0.20	16.11±9.94	29.33±18.63	21.17±13.10	3.32±1.40
5	3	√	√	√	0.27±0.06	13.13±10.51	23.75±10.61	16.97±4.38	3.01±0.96

输入帧数/N	FDR/%↓	MEA/°↓	单帧耗时/ms
5	21.34±9.52	4.52±2.15	8
10	16.97±4.38	3.01±0.96	12
15	16.21±5.10	2.85±1.02	25