基于自增强注意力机制的室内单图像分段平面三维重建

doi:10.11996/JG.j.2095-302X.2024030464

图学学报 ›› 2024, Vol. 45 ›› Issue (3): 464-471.DOI: 10.11996/JG.j.2095-302X.2024030464

• 图像处理与计算机视觉 • 上一篇下一篇

基于自增强注意力机制的室内单图像分段平面三维重建

朱光辉(), 缪君(), 胡宏利, 申基, 杜荣华

南昌航空大学航空制造工程学院，江西南昌 330063

收稿日期:2023-09-20 接受日期:2024-03-04 出版日期:2024-06-30 发布日期:2024-06-06
通讯作者:缪君(1979-)，男，副教授，博士。主要研究方向为图像处理与模式识别、三维场景重建。E-mail：miaojun@nchu.edu.cn
第一作者:朱光辉(1997-)，男，硕士研究生。主要研究方向为计算机图像处理。E-mail：2314045303@qq.com
基金资助:
国家自然科学基金项目(62162045);国家自然科学基金项目(62366032)

3D piece-wise planar reconstruction from a single indoor image based on self-augmented -attention mechanism

ZHU Guanghui(), MIAO Jun(), HU Hongli, SHEN Ji, DU Ronghua

School of Aeronautical Manufacturing Engineering, Nanchang Hangkong University, Nanchang Jiangxi 330063, China

Received:2023-09-20 Accepted:2024-03-04 Published:2024-06-30 Online:2024-06-06
First author：ZHU Guanghui (1997-), master student. His main research interest covers computer image processing. E-mail：2314045303@qq.com
Supported by:
National Natural Science Foundation of China(62162045);National Natural Science Foundation of China(62366032)

摘要/Abstract

摘要：

基于卷积神经网络(CNN)的分段平面三维重建已然成为室内场景建模研究的焦点之一。针对室内场景中，平面和非平面元素常常交织在一起，导致网络提取的平面特征中掺杂了非平面信息，从而影响了最终分割的精度；且室内场景中的平面存在尺度差异巨大的情况，带来了明显的类别不平衡，小尺度平面实例往往会失真的问题。提出了一种自增强注意力的多尺度特征融合三维分段平面重建网络，该网络能够自动学习场景中的平面特征，并有效地将不同尺度的特征信息融合，从而提升了平面实例分割的精度。同时，通过为平面实例中的每个像素分配不同的权重，特别是增加了对小尺度平面边缘像素的权重值，进一步增强了小尺度平面分割对象的通道表达。最终，采用平衡交叉熵损失和骰子损失构建了一种新的损失函数来训练模型，进一步提升了平面分割的精度。实验证明，该算法在平面召回率和分割准确度方面均取得了显著地提升，能够产生更为准确的室内三维分段平面重建模型。

关键词: 深度学习, 分段平面重建, 多尺度融合, 增强注意力, 自注意力

Abstract:

The piece-wise 3D reconstruction of indoor scenes using convolutional neural networks (CNN) has become one of the hot topics in the research of indoor scene modeling. However, the intertwining of planar and non-planar elements often leads to the network’s extraction of non-planar information mixed with planar features, thereby affecting the final segmentation accuracy. Moreover, there are significant scale differences in the planes present in indoor scenes, leading to pronounced class imbalances, where small-scale plane instances are prone to distortion. To address these challenges, this paper proposed a self-enhanced attention-based multi-scale feature fusion network for 3D plane segmentation reconstruction. This network can automatically learn planar features in the scene and effectively fuse feature information from different scales, thereby enhancing the accuracy of plane instance segmentation. At the same time, by assigning different weights to each pixel in the plane instance, particularly increasing the weight values for small-scale plane edge pixels, the channel representation of small-scale plane segmentation objects was further enhanced. Finally, a new loss function was constructed using balanced cross-entropy loss and dice loss to train the model, further improving the accuracy of plane segmentation. Extensive experiments demonstrated that the algorithm proposed achieves significant improvements in plane recall rate and segmentation accuracy, resulting in more accurate indoor 3D segmented plane reconstruction models.

Key words: deep learning, segmented plane reconstruction, multi-scale fusion, enhance attention, self-attention

中图分类号:

TP183

朱光辉, 缪君, 胡宏利, 申基, 杜荣华. 基于自增强注意力机制的室内单图像分段平面三维重建[J]. 图学学报, 2024, 45(3): 464-471.

ZHU Guanghui, MIAO Jun, HU Hongli, SHEN Ji, DU Ronghua. 3D piece-wise planar reconstruction from a single indoor image based on self-augmented -attention mechanism[J]. Journal of Graphics, 2024, 45(3): 464-471.

图/表 9

图1 算法流程

Fig. 1 Algorithm workflow

图2 自增强注意力模块

Fig. 2 Self-enhancing attention module

图3 重建效果图

Fig. 3 Reconstruction results ((a) PlaneNet; (b) PlaneAE; (c) PlaneRecNet; (d) InvPT; (e) Ours)

图4 部分分割图结果示例((a)原图；(b) PlaneNet；(c) PlaneAE；(d) PlaneRecNet；(e) InvPT；(f)本文)

Fig. 4 Examples of a partially segmented graph results ((a) Original image; (b) PlaneNet; (c) PlaneAE; (d) PlaneRecNet; (e) InvPT; (f) Ours)

表1 基于NYU-V2数据集的深度精度比较

Table 1 Comparison of depth accuracy based on NYU-V2 dataset

Method	Rel↓	Rel(sqr)↓	Log₁₀↓	RMSE_iin↓	RMSE_log↓	1.25	1.25²	1.25³
PlaneNet	0.142	0.107	0.060	0.514	0.179	81.2	95.7	98.9
PlaneAE	0.141	0.107	0.061	0.529	0.184	81.0	95.7	99.0
PlaneRecNet	0.138	0.099	0.058	0.512	0.179	82.0	95.9	99.0
InvPT	0.136	0.099	0.059	0.518	0.179	82.0	95.9	99.0
Ours	0.134	0.098	0.058	0.506	0.177	82.1	96.2	99.0

表2 基于NYU-V2数据集的分割精度比较

Table 2 Comparison of segmentation accuracy based on NYU-V2 dataset

Method	$AP m$	$AP m 50$	$AP m 75$	$AP b$	$AP b 50$	$AP b 75$
PlaneAE	45.40	57.62	46.26	49.52	61.77	49.83
PlaneRCNN	62.71	75.18	69.26	65.13	76.27	71.09
PlaneRecNet	63.88	78.95	73.17	64.93	79.00	69.80
InvPT	64.22	79.05	72.60	64.86	78.82	71.36
Ours	65.11	79.98	65.62	66.24	81.10	72.37

表2 基于NYU-V2数据集的分割精度比较

Table 2 Comparison of segmentation accuracy based on NYU-V2 dataset

Method	$AP m$	$AP m 50$	$AP m 75$	$AP b$	$AP b 50$	$AP b 75$
PlaneAE	45.40	57.62	46.26	49.52	61.77	49.83
PlaneRCNN	62.71	75.18	69.26	65.13	76.27	71.09
PlaneRecNet	63.88	78.95	73.17	64.93	79.00	69.80
InvPT	64.22	79.05	72.60	64.86	78.82	71.36
Ours	65.11	79.98	65.62	66.24	81.10	72.37

图5 ScanNet数据集((a)平面召回率；(b)像素召回率)

Fig. 5 ScanNet dataset ((a) Planar recall; (b) Pixel recall)

表3 消融实验的分割精度比较

Table 3 Comparison of segmentation accuracy in ablation experiments

Method	RI	VI	SC
原网络	0.888	1.380	0.519
SAA	0.893	1.322	0.546
L_ED	0.891	1.361	0.530
SAA+L_ED	0.911	1.304	0.551

表4 使用不同数量自注意力增强模块分割精度比较

Table 4 Comparison of segmentation accuracy using different numbers of self-attention enhancement modules

Number	RI	VI	SC
1	0.890	1.378	0.530
2	0.895	1.322	0.546
3	0.911	1.304	0.551

参考文献 24

[1]	LIU C, YANG J M, CEYLAN D, et al. PlaneNet: piece-wise planar reconstruction from a single RGB image[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 2579-2588.
[2]	YANG F T, ZHOU Z H. Recovering 3D planes from a single image via convolutional neural networks[C]// European Conference on Computer Vision. Cham: Springer, 2018: 87-103.
[3]	YE H R, XU D. Inverted pyramid multi-task transformer for dense scene understanding[C]// European Conference on Computer Vision. Cham: Springer, 2022: 514-530.
[4]	YU Z H, ZHENG J, LIAN D Z, et al. Single-image piece-wise planar 3D reconstruction via associative embedding[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 1029-1037.
[5]	LIU C, KIM K, GU J W, et al. PlaneRCNN: 3D plane detection and reconstruction from a single image[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 4445-4454.
[6]	HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 2980-2988.
[7]	QIAN Y M, FURUKAWA Y. Learning pairwise inter-plane relations for piecewise planar reconstruction[C]// European Conference on Computer Vision. Cham: Springer, 2020: 330-345.
[8]	XI W J, CHEN X J. Reconstructing piecewise planar scenes with multi-view regularization[J]. Computational Visual Media, 2019, 5(4): 337-345.
[9]	XIE Y X, RAMBACH J, SHU F W, et al. PlaneSegNet: fast and robust plane estimation using a single-stage instance segmentation CNN[C]// 2021 IEEE International Conference on Robotics and Automation. New York: IEEE Press, 2021: 13574-13580.
[10]	TAN B, XUE N, BAI S, et al. PlaneTR: structure-guided transformers for 3D plane recovery[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 4166-4175.
[11]	ZHANG W D, ZHANG Y M, SONG R, et al. 3D layout estimation via weakly supervised learning of plane parameters from 2D segmentation[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2022, 31: 868-879.
[12]	LIU J C, JI P, BANSAL N, et al. PlaneMVS: 3D plane reconstruction from multi-view stereo[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 8655-8665.
[13]	江瑞祥, 缪君, 储珺, 等. 基于多尺度聚焦网络的单图像城市场景3D平面重建[J]. 小型微型计算机系统, 2022, 43(8): 1718-1724.
	JIANG R X, MIAO J, CHU J, et al. 3D planar reconstruction of urban scene from single image based on multi-scale focusing network[J]. Journal of Chinese Computer Systems, 2022, 43(8): 1718-1724 (in Chinese).
[14]	XIE Y X, SHU F W, RAMBACH J, et al. PlaneRecNet: multi-task learning with cross-task consistency for piece-wise plane detection and reconstruction from a single RGB image[EB/OL]. (2021-10-21) [2023-08-22]. http://arxiv.org/abs/2110.11219.
[15]	HERMANN K M, KOČISKÝ T, GREFENSTETTE E, et al. Teaching machines to read and comprehend[EB/OL]. (2015-11-19) [2023-07-18]. http://arxiv.org/abs/1506.03340.
[16]	GAO Z L, XIE J T, WANG Q L, et al. Global second-order pooling convolutional networks[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2019: 3019-3028.
[17]	LEE H, KIM H E, NAM H. SRM: a style-based recalibration module for convolutional neural networks[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 1854-1862.
[18]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 7132-7141.
[19]	MNIH V, HEESS N, GRAVES A, et al. Recurrent models of visual attention[EB/OL]. (2014-06-24) [2023-06-18]. http://arxiv.org/abs/1406.6247.
[20]	HU J, SHEN L, ALBANIE S, et al. Gather-excite: exploiting feature context in convolutional neural networks[EB/OL]. (2019-01-12) [2023-07-23]. http://arxiv.org/abs/1810.12348.
[21]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]// European Conference on Computer Vision. Cham: Springer, 2018: 3-19.
[22]	WANG Z W, SHE Q, SMOLIC A. ACTION-net: multipath excitation for action recognition[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 13209-13218.
[23]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. (2017-06-12) [2023-07-13]. http://arxiv.org/abs/1706.03762.
[24]	YANG B S, WANG L Y, WONG D, et al. Convolutional self-attention networks[EB/OL]. [2023-07-13]. http://arxiv.org/abs/1904.03107

基于自增强注意力机制的室内单图像分段平面三维重建

3D piece-wise planar reconstruction from a single indoor image based on self-augmented -attention mechanism

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 24

相关文章 15

编辑推荐

Metrics

本文评价

[1]	胡欣, 常娅姝, 秦皓, 肖剑, 程鸿亮. 基于改进YOLOv8和GMM图像点集匹配的双目测距方法[J]. 图学学报, 2024, 45(4): 714-725.
[2]	牛为华, 郭迅. 基于改进YOLOv8的船舰遥感图像旋转目标检测算法[J]. 图学学报, 2024, 45(4): 726-735.
[3]	李滔, 胡婷, 武丹丹. 结合金字塔结构和注意力机制的单目深度估计[J]. 图学学报, 2024, 45(3): 454-463.
[4]	王稚儒, 常远, 鲁鹏, 潘成伟. 神经辐射场加速算法综述[J]. 图学学报, 2024, 45(1): 1-13.
[5]	王欣雨, 刘慧, 朱积成, 盛玉瑞, 张彩明. 基于高低频特征分解的深度多模态医学图像融合网络[J]. 图学学报, 2024, 45(1): 65-77.
[6]	李佳琦, 王辉, 郭宇. 基于Transformer的三角形网格分类分割网络[J]. 图学学报, 2024, 45(1): 78-89.
[7]	韩亚振, 尹梦晓, 马伟钊, 杨诗耕, 胡锦飞, 朱丛洋. DGOA：基于动态图和偏移注意力的点云上采样[J]. 图学学报, 2024, 45(1): 219-229.
[8]	王江安, 黄乐, 庞大为, 秦林珍, 梁温茜. 基于自适应聚合循环递归的稠密点云重建网络[J]. 图学学报, 2024, 45(1): 230-239.
[9]	周锐闯, 田瑾, 闫丰亭, 朱天晓, 张玉金. 融合外部注意力和图卷积的点云分类模型[J]. 图学学报, 2023, 44(6): 1162-1172.
[10]	王吉, 王森, 蒋智文, 谢志峰, 李梦甜. 基于深度条件扩散模型的零样本文本驱动虚拟人生成方法[J]. 图学学报, 2023, 44(6): 1218-1226.
[11]	杨陈成, 董秀成, 侯兵, 张党成, 向贤明, 冯琪茗. 基于参考的Transformer纹理迁移深度图像超分辨率重建[J]. 图学学报, 2023, 44(5): 861-867.
[12]	党宏社, 许怀彪, 张选德. 融合结构信息的深度学习立体匹配算法[J]. 图学学报, 2023, 44(5): 899-906.
[13]	翟永杰, 郭聪彬, 王乾铭, 赵宽, 白云山, 张冀. 基于隐含空间知识融合的输电线路多金具检测方法[J]. 图学学报, 2023, 44(5): 918-927.
[14]	杨红菊, 高敏, 张常有, 薄文, 武文佳, 曹付元. 一种面向图像修复的局部优化生成模型[J]. 图学学报, 2023, 44(5): 955-965.
[15]	毕春艳, 刘越. 基于深度学习的视频人体动作识别综述[J]. 图学学报, 2023, 44(4): 625-639.