融合结构信息的深度学习立体匹配算法

doi:10.11996/JG.j.2095-302X.2023050899

图学学报 ›› 2023, Vol. 44 ›› Issue (5): 899-906.DOI: 10.11996/JG.j.2095-302X.2023050899

• 图像处理与计算机视觉 • 上一篇下一篇

融合结构信息的深度学习立体匹配算法

党宏社¹(), 许怀彪¹, 张选德²

1.陕西科技大学电气与控制工程学院，陕西西安 710021
2.陕西科技大学电子信息与人工智能学院，陕西西安 710021

收稿日期:2023-04-28 接受日期:2023-08-01 出版日期:2023-10-31 发布日期:2023-10-31
作者简介:党宏社(1962-)，男，教授，博士。主要研究方向为工业智能控制(工业机器人)、无线传感网络和数字图象处理。E-mail：danghs@sust.edu.cn
基金资助:
国家自然科学基金项目(61871206);陕西省科技厅自然科学基金项目(2020JM-509)

Deep learning stereo matching algorithm fusing structural information

DANG Hong-she¹(), XU Huai-biao¹, ZHANG Xuan-de²

1. School of Electrical and Control Engineering, Shaanxi University of Science & Technology, Xi’an Shaanxi 710021, China
2. School of Electronic Information and Artificial Intelligence, Shaanxi University of Science & Technology, Xi’an Shaanxi 710021, China

Received:2023-04-28 Accepted:2023-08-01 Online:2023-10-31 Published:2023-10-31
About author:DANG Hong-she (1962-), professor, Ph.D. His main research interests cover industrial intelligent control (industrial robots), wireless sensor networks and digital image processing. E-mail：danghs@sust.edu.cn
Supported by:
National Natural Science Foundation of China(61871206);Natural Science Foundation Project of Shaanxi Provincial Department of Science and Technology(2020JM-509)

摘要/Abstract

摘要：

针对现有立体匹配算法在边缘区域及视差不连续区域匹配精度的不足，提出了一种融合结构信息的深度学习立体匹配算法。通过简化特征提取网络，并使用Inplace-ABN层替换BatchNorm层和激活函数层，提高卷积提取图像特征的效率；利用结合注意力机制的局部相似性模块提取图像结构特征，与卷积提取到的特征进行融合，丰富图像特征信息；计算输出特征对的相关代价和连接代价，利用相关代价卷积生成注意力权重，滤除连接代价的冗余信息，提升匹配代价计算的精确性；使用简化的沙漏网络，提升网络代价聚合的快速性。算法通过Scene Flow数据集、CREStereo数据集和KITTI数据集进行实验，实验结果表明算法的全部区域端点误差为0.45 px，对第一帧图像全部区域预测错误的比例为1.55%，预测误差大于1 px的像素比例仅有6.87%，证明所提算法相比其他算法在匹配精度上表现优秀，同时验证了算法在问题区域匹配的有效性及优势。

关键词: 深度学习, 立体匹配, 结构信息, 局部相似性模块, 匹配代价

Abstract:

To address the limitations of existing stereo matching algorithms in both edge regions and regions of discontinuous disparity, a deep learning stereo matching algorithm fusing structural information was proposed. By limiting the convolution kernel size and replacing the BatchNorm layer and activation function layer with the Inplace-ABN layer, the efficiency of convolution to extract image features was enhanced. The local similarity pattern module combined with an attention mechanism was employed to extract image structural features, and the features extracted by convolution were fused to enrich image feature information. The correlation volume and connection volume of the output feature were calculated. By utilizing the correlation volume to generate attention weights, the algorithm filtered out the redundant information of the connection volume and improved the accuracy of the stereo matching cost volume. In order to expedite network cost aggregation, a simplified hourglass network was employed. The algorithm was tested against the Scene Flow dataset, CREStereo dataset, and KITTI dataset. The experimental results demonstrated that the algorithm had an overall region endpoint error of 0.45 pixels. In the first frame image, only 1.55% of regions were incorrectly predicted, and merely 6.87% of pixels exhibited prediction errors greater than 1 pixel. These results demonstrated the excellent performance of the proposed algorithm compared to other algorithms in terms of matching accuracy. Furthermore, it validated the effectiveness and advantages of the algorithm in matching problematic areas.

Key words: deep learning, stereo matching, structural information, local similarity pattern module, cost volume

中图分类号:

TP751.1

党宏社, 许怀彪, 张选德. 融合结构信息的深度学习立体匹配算法[J]. 图学学报, 2023, 44(5): 899-906.

DANG Hong-she, XU Huai-biao, ZHANG Xuan-de. Deep learning stereo matching algorithm fusing structural information[J]. Journal of Graphics, 2023, 44(5): 899-906.

图/表 11

参考文献 21

[1]	李云龙, 卿粼波, 韩龙玫, 等. 视觉可供性研究综述[J]. 计算机工程与应用, 2022, 58(18): 1-15. DOI
	LI Y L, QING L B, HAN L M, et al. Survey on visual affordance research[J]. Computer Engineering and Applications, 2022, 58(18): 1-15. (in Chinese) DOI
[2]	陈炎, 杨丽丽, 王振鹏. 双目视觉的匹配算法综述[J]. 图学学报, 2020, 41(5): 702-708. DOI
	CHEN Y, YANG L L, WANG Z P. Literature survey on stereo vision matching algorithms[J]. Journal of Graphics, 2020, 41(5): 702-708. (in Chinese) DOI
[3]	尹晨阳, 职恒辉, 李慧斌. 基于深度学习的双目立体匹配方法综述[J]. 计算机工程, 2022, 48(10): 1-12. DOI
	YIN C Y, ZHI H H, LI H B. Survey of binocular stereo-matching methods based on deep learning[J]. Computer Engineering, 2022, 48(10): 1-12. (in Chinese) DOI
[4]	MAYER N, ILG E, HÄUSSER P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 4040-4048.
[5]	CHANG J R, CHEN Y S. Pyramid stereo matching network[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 5410-5418.
[6]	GUO X Y, YANG K, YANG W K, et al. Group-wise correlation stereo network[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 3268-3277.
[7]	XU H F, ZHANG J Y. AANet: adaptive aggregation network for efficient stereo matching[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1956-1965.
[8]	CHENG X L, ZHONG Y R, HARANDI M, et al. Hierarchical neural architecture search for deep stereo matching[C]// The 34th International Conference on Neural Information Processing Systems. New York: ACM, 2020: 22158-22169.
[9]	LI J K, WANG P S, XIONG P F, et al. Practical stereo matching via cascaded recurrent network with adaptive correlation[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 16242-16251.
[10]	LIPSON L, TEED Z, DENG J. RAFT-stereo: multilevel recurrent field transforms for stereo matching[C]// 2021 International Conference on 3D Vision. New York: IEEE Press, 2022: 218-227.
[11]	LIU B Y, YU H M, LONG Y Q. Local similarity pattern and cost self-reassembling for deep stereo matching networks[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(2): 1647-1655. DOI URL
[12]	NEWELL A, YANG K Y, DENG J. Stacked hourglass networks for human pose estimation[M]// Computer Vision - ECCV 2016. Cham: Springer International Publishing, 2016: 483-499.
[13]	李瞳, 马伟, 徐士彪, 等. 适应立体匹配任务的端到端深度网络[J]. 计算机研究与发展, 2020, 57(7): 1531-1538.
	LI T, MA W, XU S B, et al. Task-adaptive end-to-end networks for stereo matching[J]. Journal of Computer Research and Development, 2020, 57(7): 1531-1538. (in Chinese)
[14]	KOUTINI K, EGHBAL-ZADEH H, DORFER M, et al. The receptive field as a regularizer in deep convolutional neural networks for acoustic scene classification[C]// The 27th European Signal Processing Conference. New York: IEEE Press, 2019: 1-5.
[15]	TAN M, LE Q V. MixConv: mixed depthwise convolutional kernels"[EB/OL]. [2023-01-18]. https://arxiv.org/abs/1907.09595.
[16]	BULÒ S R, PORZI L, KONTSCHIEDER P. In-place activated BatchNorm for memory-optimized training of DNNs[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 5639-5647.
[17]	RIDNIK T, LAWEN H, NOY A, et al. TResNet: high performance GPU-dedicated architecture[C]// 2021 IEEE Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2021: 1399-1408.
[18]	WANG Y N, GU M J, ZHU Y F, et al. Improvement of AD-census algorithm based on stereo vision[J]. Sensors, 2022, 22(18): 6933. DOI URL
[19]	XU G W, CHENG J D, GUO P, et al. Attention concatenation volume for accurate and efficient stereo matching[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 12971-12980.
[20]	KENDALL A, MARTIROSYAN H, DASGUPTA S, et al. End-to-end learning of geometry and context for deep stereo regression[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 66-75.
[21]	GEIGER A, LENZ P, STILLER C, et al. Vision meets robotics: the KITTI dataset[J]. International Journal of Robotics Research, 2013, 32(11): 1231-1237. DOI URL

Layer	Kernal size，Channel	Output
Conv0_1	3×3，32	$\frac{1}{2}H\times \frac{1}{2}W\times 32$
Conv0_2	1×1，32	$\frac{1}{2}H\times \frac{1}{2}W\times 32$
Conv1_x	3×3，32 1×1，32	$\frac{1}{2}H\times \frac{1}{2}W\times 32$
Conv2_x+SE	3×3，64 1×1，64	$\frac{1}{4}H\times \frac{1}{4}W\times 64$
Conv3_x+SE	3×3，128 1×1，128	$\frac{1}{4}H\times \frac{1}{4}W\times 128$
Conv4_x+SE	3×3，128 1×1，128	$\frac{1}{4}H\times \frac{1}{4}W\times 128$

Layer	Kernal size，Channel	Output
Conv0_1	3×3，32	$\frac{1}{2}H\times \frac{1}{2}W\times 32$
Conv0_2	1×1，32	$\frac{1}{2}H\times \frac{1}{2}W\times 32$
Conv1_x	3×3，32 1×1，32	$\frac{1}{2}H\times \frac{1}{2}W\times 32$
Conv2_x+SE	3×3，64 1×1，64	$\frac{1}{4}H\times \frac{1}{4}W\times 64$
Conv3_x+SE	3×3，128 1×1，128	$\frac{1}{4}H\times \frac{1}{4}W\times 128$
Conv4_x+SE	3×3，128 1×1，128	$\frac{1}{4}H\times \frac{1}{4}W\times 128$

Method	EPE (px)	D1-all (%)	>1 px (%)	Time (s)
1. ResCNN	1.07	3.89	7.68	0.73
2. Improved ResCNN	1.02	3.54	7.65	0.67
3. Improved ResCNN + LSP	0.78	1.67	7.31	0.70
4. Improved ResCNN + ACV	0.51	1.95	7.03	0.60
5. Improved ResCNN + LSP + ACV	0.45	1.55	6.87	0.62

Method	EPE (px)	D1-all (%)	>1 px (%)	Time (s)
1. ResCNN	1.07	3.89	7.68	0.73
2. Improved ResCNN	1.02	3.54	7.65	0.67
3. Improved ResCNN + LSP	0.78	1.67	7.31	0.70
4. Improved ResCNN + ACV	0.51	1.95	7.03	0.60
5. Improved ResCNN + LSP + ACV	0.45	1.55	6.87	0.62

Method	KITTI2015
Method	D1-bg	D1-fg	D1-all
PSMNet^[5]	1.86	4.62	2.32
GwcNet^[6]	1.71	3.93	2.11
AANet^[7]	1.65	3.96	2.03
LEAStereo^[8]	1.40	2.91	1.65
CREStereo^[9]	1.45	2.86	1.69
ACVNet^[19]	1.37	3.07	1.65
RAFT-Stereo^[10]	1.58	3.05	1.82
ILANet	1.38	2.98	1.61

融合结构信息的深度学习立体匹配算法

Deep learning stereo matching algorithm fusing structural information

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 11

参考文献 21

相关文章 15

编辑推荐

Metrics

本文评价

Method	Out-Noc	Out-All	Avg-Noc (px)	Avg-All (px)
PSMNet^[5]	1.49	1.89	0.5	0.6
GwcNet^[6]	1.32	1.70	0.5	0.5
AANet^[7]	1.91	2.42	0.5	0.6
LEAStereo^[8]	1.13	1.45	0.5	0.5
CREStereo^[9]	1.14	1.46	0.4	0.5
ACVNet^[19]	1.13	1.47	0.4	0.5
RAFT-Stereo^[10]	1.30	1.66	0.4	0.5
ILANet	1.10	1.46	0.4	0.5

Method	EPE (px)	D1-all (%)	>1 px (%)
GwcNet	0.76	2.71	8.01
PSMNet	1.09	3.89	7.88
ACVNet	0.48	1.59	7.06
ILANet	0.44	1.53	6.89

[1]	周锐闯, 田瑾, 闫丰亭, 朱天晓, 张玉金 . 融合外部注意力和图卷积的点云分类模型 [J]. 图学学报, 2023, 44(6): 1162-1172.
[2]	王吉, 王森, 蒋智文, 谢志峰, 李梦甜. 基于深度条件扩散模型的零样本文本驱动虚拟人生成方法 [J]. 图学学报, 2023, 44(6): 1218-1226.
[3]	杨陈成, 董秀成, 侯兵, 张党成, 向贤明, 冯琪茗. 基于参考的Transformer纹理迁移深度图像超分辨率重建[J]. 图学学报, 2023, 44(5): 861-867.
[4]	翟永杰, 郭聪彬, 王乾铭, 赵宽, 白云山, 张冀. 基于隐含空间知识融合的输电线路多金具检测方法[J]. 图学学报, 2023, 44(5): 918-927.
[5]	杨红菊, 高敏, 张常有, 薄文, 武文佳, 曹付元. 一种面向图像修复的局部优化生成模型[J]. 图学学报, 2023, 44(5): 955-965.
[6]	毕春艳, 刘越. 基于深度学习的视频人体动作识别综述[J]. 图学学报, 2023, 44(4): 625-639.
[7]	曹义亲, 周一纬, 徐露. 基于E-YOLOX的实时金属表面缺陷检测算法[J]. 图学学报, 2023, 44(4): 677-690.
[8]	邵俊棋, 钱文华, 徐启豪. 基于条件残差生成对抗网络的风景图生成[J]. 图学学报, 2023, 44(4): 710-717.
[9]	余伟群, 刘佳涛, 张亚萍. 融合注意力的拉普拉斯金字塔单目深度估计[J]. 图学学报, 2023, 44(4): 728-738.
[10]	郭印宏, 王立春, 李爽. 基于重复性和特异性约束的图像特征匹配[J]. 图学学报, 2023, 44(4): 739-746.
[11]	毛爱坤, 刘昕明, 陈文壮, 宋绍楼. 改进YOLOv5算法的变电站仪表目标检测方法[J]. 图学学报, 2023, 44(3): 448-455.
[12]	王佳婧, 王晨, 朱媛媛, 王笑梅. 基于民国纸币的图元素匹配检索[J]. 图学学报, 2023, 44(3): 492-501.
[13]	杨柳, 吴晓群. 基于深度学习的三维形状补全研究综述[J]. 图学学报, 2023, 44(2): 201-215.
[14]	曾武, 朱恒亮, 邢树礼, 林江宏, 毛国君. 显著性检测引导的图像数据增强方法[J]. 图学学报, 2023, 44(2): 260-270.
[15]	罗启明, 吴昊, 夏信, 袁国武. 基于Dual Dense U-Net的云南壁画破损区域预测[J]. 图学学报, 2023, 44(2): 304-312.