全景图像视频的场景分析与内容处理方法综述

doi:10.11996/JG.j.2095-302X.2023040640

图学学报 ›› 2023, Vol. 44 ›› Issue (4): 640-657.DOI: 10.11996/JG.j.2095-302X.2023040640

全景图像视频的场景分析与内容处理方法综述

谢红霞¹(), 胡毓宁¹, 张赟²(), 王亚奇², 杜辉², 秦爱红²

1.浙大城市学院计算机与计算科学学院，浙江杭州 310015
2.浙江传媒学院媒体工程学院，浙江杭州 310018

收稿日期:2023-02-03 接受日期:2023-04-11 出版日期:2023-08-31 发布日期:2023-08-16
通讯作者: 张赟(1984-)，男，教授，博士。主要研究方向为计算机图形学、虚拟现实等。E-mail：zhangyun@cuz.edu.cn
作者简介:
谢红霞(1971-)，女，讲师，硕士。主要研究方向为计算机技术应用、虚拟现实。E-mail：xiehx@zucc.edu.cn
基金资助:
浙江省基础公益研究计划项目(LGG22F020009);浙江省基础公益研究计划项目(LGF21F020002);浙江省基础公益研究计划项目(LGF22F020015);国家自然科学基金项目(62206242);浙江省影视媒体技术研究重点实验室开放课题(2020E10015);浙江传媒学院2021年第十六批教学改革项目(jgxm202131)

Survey of methods for scene analysis and content processing in panoramic images and videos

XIE Hong-xia¹(), HU Yu-ning¹, ZHANG Yun²(), WANG Ya-qi², DU Hui², QIN Ai-hong²

1. School of Computer & Computing Science, Hangzhou City University, Hangzhou Zhejiang 310015, China
2. College of Media Engineering, Communication University of Zhejiang, Hangzhou Zhejiang 310018, China

Received:2023-02-03 Accepted:2023-04-11 Online:2023-08-31 Published:2023-08-16
Contact: ZHANG Yun (1984-), professor, Ph.D. His main research interests cover computer graphics, virtue reality, etc. E-mail：zhangyun@cuz.edu.cn
About author:
XIE Hong-xia (1971-), lecturer, master. Her main research interests cover computer technology application, virtue reality. E-mail：xiehx@zucc.edu.cn
Supported by:
Zhejiang Province Public Welfare Technology Application Research(LGG22F020009);Zhejiang Province Public Welfare Technology Application Research(LGF21F020002);Zhejiang Province Public Welfare Technology Application Research(LGF22F020015);National Natural Science Foundation of China(62206242);Key Laboratory of Film and TV Media Technology of Zhejiang Province(2020E10015);The 16th Teaching Reform Project in 2021 of Communication University of Zhejiang(jgxm202131)

摘要/Abstract

摘要：

近年来，随着全景内容获取和交互的软硬件技术的快速发展，全景图像视频的数量激增，如何对全景内容进行高质量地分析和处理越来越成为虚拟现实领域的研究热点。当前，全景内容分析和处理在理论和应用方面面临着巨大挑战，关于该领域的关键问题在已有文献中未见系统全面地总结和研究。为了更好地促进该领域的研究和应用推广，针对全景图像视频的场景分析与内容处理近期的主要工作进行综述。在全景场景分析方面，分析了全景图像视频的深度学习网络、深度恢复、重要性检测、目标检测的研究工作；在全景内容处理方面，分析了全景图像视频的交互式浏览、去抖和校正、内容编辑的研究工作。最后，对综述进行了总结，并展望了未来在立体视图下全景图像视频的场景分析与内容处理方面的研究趋势。

关键词: 虚拟现实, 全景图像视频, 场景分析, 内容处理, 立体视图

Abstract:

In recent years, the rapid development of software and hardware technologies for acquiring and interacting with panoramic content has led to a significant increase in the number of panoramic images and videos. Immersive media with 360-degree panoramic images and videos as the main content has been widely used in the field of virtual reality and enhancement implementation. Compared with traditional 2D images and videos, panoramic images and videos can provide users with a new immersive experience. With wearable devices, users can freely watch the content from all perspectives through head movement. At present, the number of panoramic images and videos has soared, but it is usually difficult to obtain satisfactory panoramic images and videos, due to the difficulty in obtaining panoramic content and the lack of effective editing tools. Therefore, analyzing and processing panoramic content with high quality has become an increasingly important research topic in the field of virtual reality. However, both in theory and application, the analysis and processing of panoramic content face significant challenges. Despite this, there is a lack of systematic and comprehensive summaries and research on the key issues in this field in existing literature. In order to better promote research and application in this area, a survey was provided on the recent works of scene analysis and content processing of panoramic images and videos. In terms of panoramic scene analysis, this survey reviewed the research on depth learning networks, depth recovery, importance detection, and target detection for panoramic images and videos. In terms of panoramic content processing, the survey analyzed the research on interactive browsing, stabilization and correction, and content editing of panoramic image video. Finally, the overview was summarized, with an outlook on future research trends in scene analysis and content processing of panoramic images and videos under the stereo view.

Key words: virtual reality, panoramic images and videos, scene analysis, content processing, stereoscopic views

中图分类号:

TP391

谢红霞, 胡毓宁, 张赟, 王亚奇, 杜辉, 秦爱红. 全景图像视频的场景分析与内容处理方法综述[J]. 图学学报, 2023, 44(4): 640-657.

XIE Hong-xia, HU Yu-ning, ZHANG Yun, WANG Ya-qi, DU Hui, QIN Ai-hong. Survey of methods for scene analysis and content processing in panoramic images and videos[J]. Journal of Graphics, 2023, 44(4): 640-657.

图/表 18

图1 最新穿戴式设备的用户体验效果图[1-2]及360°全景图像的不同平面投影方式

Fig. 1 User experience renderings of latest wearable devices[1-2] and different planar projections of 360° panoramas

表1 全景图像视频场景分析研究工作总结

Table 1 Summary of research work on scene analysis of panoramic images and videos

研究大类	研究内容	代表文献	主要特点
1全景图像视频的场景分析	1.1全景图像视频的深度学习网络	文献[14-15]	基于立方体投影表示
		文献[16-17]	基于球面的投影表示
		文献[18]	基于等矩形的投影表示
	1.2场景深度恢复	文献[24-26]	单目深度恢复——基于卷积神经网络
		文献[33-34]	双目深度恢复——基于全局优化的方法
		文献[35-38]	双目深度恢复——基于深度学习的方法
	1.3场景重要性检测	文献[45-46]	全景图像重要性检测——基于平面投影
		文献[48]	全景图像重要性检测——基于球面
		文献[51]	全景视频重要性检测——基于立方体填充
		文献[52-53]	全景视频重要性检测——基于球面卷积
		文献[55]	全景视频重要性检测——基于视点预测
	1.4场景目标检测	文献[60-61]	基于单一投影的方法
		文献[62-63]	基于视野并交集的方法
		文献[64-65]	基于多种投影的方法

图2 360°全景图扭曲的可视化表示[9] ((a) 360°全景图像；(b)具有Tissot指示的等矩形图像)

Fig. 2 Visualization of distortions in 360° panoramas[9] (a) Spherical 360° image; (b) The equirectangular image with Tissot’s indicatrices)

图3 基于立方体、球面和等矩形平面表示的CNN ((a)立方体投影表示[14]；(b)球面多面体表示[16]；(c)等矩形表示[18])

Fig. 3 CNN based on cube map, sphere and equirectangular representations ((a) Cubmap representation[14]; (b) SpherePHD representation[16]; (c) Equirectangular representation[18])

图4 根据图像扭曲模型对采样网格进行变形，从而对感受视野进行修正[4] ((a)输入特征图；(b)输出特征图)

Fig. 4 The sampling grid is deformed according to the image distortion model, so that the receptive field is rectified[4] ((a) Input feature map; (b) Output feature map)

图5 提出基于切向块的方法来消除全景扭曲，并通过令牌流更好地适配室内场景几何结构[26]

Fig. 5 The tangent-patch is proposed to remove panoramic distortions, and the token flows force the token positions to fit the structure of the sofa better[26]

图6 360°立体全景的深度估计网络[38] ((a)基于双分支的特征提取；(b)用于扩大感受野的ASPP模块；(c)用于非线性球面投影的学习代价计算)

Fig. 6 360° stereo depth estimation network[38] ((a) Two-branch feature extractor; (b) The ASPP module to enlarge the receptive field; (c) The learnable cost volume to account for the nonlinear spherical projection)

图7 360°图像重要性检测方法流程图[47]

Fig. 7 Flowchart of saliency detection of 360° images[47]

图8 360°视频重要性预测的网络模型[55]

Fig. 8 Network for the saliency prediction of 360° images[55]

图9 2种主要的将CNN应用于360°图像的策略分析[5]

Fig. 9 Two existing strategies for applying CNNs to 360° images[5]

表2 全景图像视频内容处理研究工作总结

Table 2 Summary of research work on content processing of panoramic images and videos

研究大类	研究内容	代表文献	主要特点
2全景图像视频的内容处理	2.1交互式浏览	文献[72-73]	自动视频导航
		文献[75-77]	交互式和可视化浏览
	2.2去抖和校正	文献[78]	自动全景图像校正
		文献[79-80]	全景视频去抖
		文献[81]	全景视频去抖+校正
	2.3内容编辑	文献[83-85]	全景内容补全
		文献[87-88]	全景克隆与颜色编辑
		文献[89-90]	内容增强——全景图超分辨率

图10 交互式和自动的360°视频导航[73] ((a) 360°视频；(b)由(a)转换来的正常视野的视频；(c)由(a)转换到(b)的相机路径)

Fig. 10 Interactive and automatic 360° video navigation [73] ((a) 360° video; (b) Normal field-of-view (NFoV) video converted from (a); (c) Camera path for converting (a) to (b))

图11 360°视频的协同浏览[77]

Fig. 11 Collaborative navigation of 360° videos[77]

图12 360°手持相机、全景视频以及视频去抖前后的运动轨迹[79] ((a)手持360°相机；(b) 360°×180°的全球面视频；(c)原始的运动轨迹；(d)基于旋转的去抖；(e)基于变形旋转的去抖)

Fig. 12 Hand-held 360° camera, panoramic video, and the tracked motion before and after stabilization[79] ((a) Hand-held 360° cameras; (b) Full-spherical 360°×180° video; (c) Original motion tracks; (d) Pure rotation stabilization; (e) Deformed rotation stabilization)

图13 ERP视角注意的全景图像补全[85]

Fig. 13 Viewport-aware inpainting based on the ERP image[85]

图14 360°视频内容补全流程图[86]

Fig. 14 Flowchart of 360° video completion[86]

图15 基于RBF插值的360°全景图编辑扩散流程[88] ((a)输入图像；(b)颜色量化；(c)适应性采样；(d)径向基函数插值；(e)结果渲染；(f)球面和放大视图)

Fig. 15 360° panorama edit propagation based on the RBF interpolation[88] ((a) Input image; (b) Color quantization; (c) Adaptive sampling; (d) RBF interpolation; (e) Result rendering; (f) Spherical & Zoom-in views)

图16 SphereSR 的总体框架[91]

Fig. 16 Overall framework of the proposed SphereSR[91]

参考文献 94

[1]	HTC. VIVE XR Elite[EB/OL]. (2022-12-10]) [2023-01-02]. https://www.vive.com/uk/product/vive-xr-elite/overview/.
[2]	Meta. Quest Pro[EB/OL]. (2022-11-10) [2023-01-02]. https://www.meta.com/gb/quest/quest-pro/.
[3]	SHEN Z J, LIN C Y, NIE L, et al. Distortion-tolerant monocular depth estimation on omnidirectional images using dual-cubemap[C]// 2021 IEEE International Conference on Multimedia and Expo. New York: IEEE Press, 2021: 1-6.
[4]	TATENO K, NAVAB N, TOMBARI F. Distortion-aware convolutional filters for dense prediction in panoramic images[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 732-750.
[5]	SU Y C, GRAUMAN K. Learning spherical convolution for fast features from 360° imagery[EB/OL]. [2022-12-02]. https://arxiv.org/abs/1708.00919.
[6]	COORS B, CONDURACHE A P, GEIGER A. SphereNet: learning spherical representations for detection and classification in omnidirectional images[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 525-541.
[7]	WANG M, LI Y J, ZHANG W X, et al. Transitioning360: content-aware NFoV virtual camera paths for 360° video playback[C]// 2020 IEEE International Symposium on Mixed and Augmented Reality. New York: IEEE Press, 2020: 185-194.
[8]	LI Y J, SHI J C, ZHANG F L, et al. Bullet comments for 360° video[C]// 2022 IEEE Conference on Virtual Reality and 3D User Interfaces. New York: IEEE Press, 2022: 1-10.
[9]	ZHANG Y, ZHANG F L, LAI Y K, et al. Efficient propagation of sparse edits on 360^∘ panoramas[J]. Computers & Graphics, 2021, 96: 61-70. DOI URL
[10]	WANG W G, LAI Q X, FU H Z, et al. Salient object detection in the deep learning era: an In-depth survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6): 3239-3259. DOI URL
[11]	BAI S J, GENG Z Y, SAVANI Y, et al. Deep equilibrium optical flow estimation[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 610-620.
[12]	GUO M H, LU C Z, HOU Q B, et al. SegNeXt: rethinking convolutional attention design for semantic segmentation[EB/OL]. [2022-12-02]. https://arxiv.org/abs/2209.08575.
[13]	WANG W G, SHEN J B, XIE J W, et al. Revisiting video saliency prediction in the deep learning era[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(1): 220-237. DOI URL
[14]	XIONG B, GRAUMAN K. Snap angle prediction for 360° panoramas[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 3-20.
[15]	WOO HAN S, YOUNG SUH D. A 360-degree panoramic image inpainting network using a cube map[J]. Computers, Materials & Continua, 2020, 66(1): 213-228.
[16]	LEE Y, JEONG J, YUN J, et al. SpherePHD: applying CNNs on a spherical PolyHeDron representation of 360° images[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 9173-9181.
[17]	EDER M, SHVETS M, LIM J, et al. Tangent images for mitigating spherical distortion[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 12423-12431.
[18]	SU Y C, GRAUMAN K. Kernel transformer networks for compact spherical convolution[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 9434-9443.
[19]	BHOI A. Monocular depth estimation: a survey[EB/OL]. (2019-01-27) [2022-12-23]. https://arxiv.org/abs/1901.09402.
[20]	ZHANG Y R, GONG M G, LI J Z, et al. Self-supervised monocular depth estimation with multiscale perception[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2022, 31: 3251-3266. DOI URL
[21]	LEI Z Y, WANG Y, LI Z J, et al. Attention based multilayer feature fusion convolutional neural network for unsupervised monocular depth estimation[J]. Neurocomputing, 2021, 423: 343-352. DOI URL
[22]	MING Y, MENG X Y, FAN C X, et al. Deep learning for monocular depth estimation: a review[J]. Neurocomputing, 2021, 438: 14-33. DOI URL
[23]	WANG F E, YEH Y H, SUN M, et al. BiFuse:monocular 360 depth estimation via Bi-projection fusion[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 459-468.
[24]	JIN L, XU Y Y, ZHENG J, et al. Geometric structure based and regularized depth estimation from 360 indoor imagery[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 886-895.
[25]	PINTORE G, AGUS M, ALMANSA E, et al. SliceNet: deep dense depth estimation from a single indoor panorama using a slice-based representation[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 11531-11540.
[26]	SHEN Z J, LIN C Y, LIAO K, et al. PanoFormer: panorama transformer for indoor 360° depth estimation[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2022: 195-211.
[27]	LI M, WANG S B, YUAN W H, et al. S²Net: accurate panorama depth estimation on spherical surface[J]. IEEE Robotics and Automation Letters, 2023, 8(2): 1053-1060. DOI URL
[28]	REY-AREA M, YUAN M Z, RICHARDT C. 360MonoDepth: high-resolution 360° monocular depth estimation[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 3752-3762.
[29]	PENG C H, ZHANG J Y. High-resolution depth estimation for 360° panoramas through perspective and panoramic depth images registration[C]// 2023 IEEE/CVF Winter Conference on Applications of Computer Vision. New York: IEEE Press, 2023: 3115-3124.
[30]	WANG L, YANG R G. Global stereo matching leveraged by sparse ground control points[C]// 2011 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2011: 3033-3040.
[31]	YUAN W M, MENG C, TONG X Y, et al. Efficient local stereo matching algorithm based on fast gradient domain guided image filtering[J]. Signal Processing: Image Communication, 2021, 95: 116280. DOI URL
[32]	YANG Q X. Hardware-efficient bilateral filtering for stereo matching[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(5): 1026-1032. DOI PMID
[33]	LI S G. Binocular spherical stereo[J]. IEEE Transactions on Intelligent Transportation Systems, 2008, 9(4): 589-600. DOI URL
[34]	KIM H, HILTON A. 3D scene reconstruction from multiple spherical stereo pairs[J]. International Journal of Computer Vision, 2013, 104(1): 94-116. DOI URL
[35]	ŽBONTAR J, LECUN Y. Computing the stereo matching cost with a convolutional neural network[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2015: 1592-1599.
[36]	CHEN B L, JUNG C. Patch-based stereo matching using 3D convolutional neural networks[C]// The 25th IEEE International Conference on Image Processing. New York: IEEE Press, 2018: 3633-3637.
[37]	CHANG J R, CHEN Y S. Pyramid stereo matching network[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 5410-5418.
[38]	WANG N H, SOLARTE B, TSAI Y H, et al. 360SD-net: 360° stereo depth estimation with learnable cost volume[C]// 2020 IEEE International Conference on Robotics and Automation. New York: IEEE Press, 2020: 582-588.
[39]	WEGNER K, STANKIEWICZ O, GRAJEK T, et al. Depth estimation from stereoscopic 360-degree video[C]// The 25th IEEE International Conference on Image Processing. New York: IEEE Press, 2018: 2945-2948.
[40]	BORJI A, CHENG M M, HOU Q B, et al. Salient object detection: a survey[J]. Computational Visual Media, 2019, 5(2): 117-150. DOI
[41]	WU Y H, LIU Y, ZHANG L, et al. EDN: salient object detection via extremely-downsampled network[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2022, 31: 3125-3136. DOI URL
[42]	张璐. 基于深度特征融合的显著目标检测算法研究[D]. 大连: 大连理工大学, 2021.
	ZHANG L. Research on salient object detection algorithms based on deep feature integration[D]. Dalian: Dalian University of Technology, 2021 (in Chinese).
[43]	WANG W G, SHEN J B, XIE J W, et al. Revisiting video saliency prediction in the deep learning era[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(1): 220-237. DOI URL
[44]	LIU J J, LIU Z A, PENG P, et al. Rethinking the U-shape structure for salient object detection[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2021, 30: 9030-9042. DOI URL
[45]	MONROY R, LUTZ S, CHALASANI T, et al. SalNet360: Saliency maps for omni-directional images with CNN[J]. Signal Processing: Image Communication, 2018, 69: 26-34. DOI URL
[46]	LI J, SU J M, XIA C Q, et al. Distortion-adaptive salient object detection in 360° omnidirectional images[J]. IEEE Journal of Selected Topics in Signal Processing, 2020, 14(1): 38-48. DOI URL
[47]	MA G X, LI S, CHEN C, et al. Stage-wise salient object detection in 360° omnidirectional image via object-level semantical saliency ranking[J]. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(12): 3535-3545. DOI URL
[48]	LV H R, YANG Q, LI C L, et al. SalGCN: saliency prediction for 360-degree images based on spherical graph convolutional networks[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 682-690.
[49]	ZHANG R P, CHEN C Y, ZHANG J C, et al. 360-degree visual saliency detection based on fast-mapped convolution and adaptive equator-bias perception[J]. The Visual Computer, 2023, 39(3): 1163-1180. DOI
[50]	GAO P, CHEN X L, QUAN R, et al. MRGAN360: multi-stage recurrent generative adversarial network for 360 degree image saliency prediction[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2303.08525.
[51]	CHENG H T, CHAO C H, DONG J D, et al. Cube padding for weakly-supervised saliency prediction in 360° videos[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2018: 1420-1429.
[52]	ZHANG Z H, XU Y Y, YU J Y, et al. Saliency detection in 360° videos[C]// European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 504-520.
[53]	QIAO M L, XU M, WANG Z L, et al. Viewport-dependent saliency prediction in 360° video[J]. IEEE Transactions on Multimedia, 2021, 23: 748-760. DOI URL
[54]	DU R F, VARSHNEY A. Saliency computation for virtual cinematography in 360° videos[J]. IEEE Computer Graphics and Applications, 2021, 41(4): 99-106. DOI PMID
[55]	BERNAL-BERDUN E, MARTIN D, GUTIERREZ D, et al. SST-sal: a spherical spatio-temporal approach for saliency prediction in 360° videos[J]. Computers & Graphics, 2022, 106, C: 200-209.
[56]	YUN H, LEE S H, KIM G. Panoramic vision transformer for saliency detection in 360° videos[M]// Lecture Notes in Computer Science. Cham: Springer Nature Switzerland, 2022: 422-439.
[57]	ZHANG D W, HAN J W, CHENG G, et al. Weakly supervised object localization and detection: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(9): 5866-5885.
[58]	LAI W S, HUANG Y J, JOSHI N, et al. Semantic-driven generation of hyperlapse from 360 degree video[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(9): 2610-2621. DOI URL
[59]	XIAO J X, EHINGER K A, OLIVA A, et al. Recognizing scene viewpoint using panoramic place representation[C]// 2012 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2012: 2695-2702.
[60]	YANG W, QIAN Y, KÄMÄRÄINEN J K, et al. Object detection in equirectangular panorama[C]// The 24th International Conference on Pattern Recognition. New York: IEEE Press, 2018: 2190-2195.
[61]	WANG K H, LAI S H. Object detection in curved space for 360-degree camera[C]// 2019 IEEE International Conference on Acoustics, Speech and Signal Processing. New York: IEEE Press, 2019: 3642-3646.
[62]	ZHAO P Y, YOU A S, ZHANG Y X, et al. Spherical criteria for fast and accurate 360° object detection[J]. The AAAI Conference on Artificial Intelligence, 2020, 34(7): 12959-12966.
[63]	CAO M, IKEHATA S, AIZAWA K. Field-of-view IoU for object detection in 360° images[EB/OL]. [2022-12-12]. https://arxiv.org/abs/2202.03176.
[64]	ZHENG Z S, LIN C Y, NIE L, et al. Bi-projection for 360° image object detection bridged by RoI searcher[J]. Journal of Visual Communication and Image Representation, 2022, 89: 103660. DOI URL
[65]	CAO M, IKEHATA S, AIZAWA K. Dual-erp representation for object detection in 360° images[C]// 2022 IEEE International Conference on Image Processing. New York: IEEE Press, 2022: 2016-2020.
[66]	WIEN M, BOYCE J M, STOCKHAMMER T, et al. Standardization status of immersive video coding[J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2019, 9(1): 5-17. DOI
[67]	JPEG Requirements Subgroup. JPEG 360 metadata use cases[EB/OL]. (2018-02-02) [2023-01-02]. https://jpeg.org/downloads/jpeg360/JPEG360-use-cases.pdf.
[68]	XU M, LI C, ZHANG S Y, et al. State-of-the-art in 360° video/image processing: perception, assessment and compression[J]. IEEE Journal of Selected Topics in Signal Processing, 2020, 14(1): 5-26. DOI URL
[69]	GUTIÉRREZ J, DAVID E, RAI Y, et al. Toolbox and dataset for the development of saliency and scanpath models for omnidirectional/360° still images[J]. Signal Processing: Image Communication, 2018, 69: 35-42. DOI URL
[70]	CHEN J Y, LUO Z X, WANG Z L, et al. Live360: viewport-aware transmission optimization in live 360-degree video streaming[J]. IEEE Transactions on Broadcasting, 2023, 69(1): 85-96. DOI URL
[71]	HU M, WANG L F, TAN B, et al. Two-tier 360-degree video delivery control in multiuser immersive communications systems[J]. IEEE Transactions on Vehicular Technology, 2023, 72(3): 4119-4123. DOI URL
[72]	HU H N, LIN Y C, LIU M Y, et al. Deep 360 pilot: learning a deep agent for piloting through 360° sports videos[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1396-1405.
[73]	KANG K, CHO S. Interactive and automatic navigation for 360° video playback[J]. ACM Transactions on Graphics, 2019, 38(4): 108:1-108:11.
[74]	IRFAN M, MUHAMMAD K, SAJJAD M, et al. Deepview: deep-learning-based users field of view selection in 360° videos for industrial environments[J]. IEEE Internet of Things Journal, 2023, 10(4): 2903-2912. DOI URL
[75]	PAVEL A, HARTMANN B, AGRAWALA M. Shot orientation controls for interactive cinematography with 360 video[C]// The 30th Annual ACM Symposium on User Interface Software and Technology. New York: ACM, 2017: 289-297.
[76]	WALLGRÜN J O, BAGHER M M, SAJJADI P, et al. A comparison of visual attention guiding approaches for 360° image-based VR tours[C]// 2020 IEEE Conference on Virtual Reality and 3D User Interfaces. New York: IEEE Press, 2020: 83-91.
[77]	KUMAR K, PORETSKI L, LI J N, et al. Tourgether360: collaborative exploration of 360° videos using pseudo-spatial navigation[J]. Proceedings of the ACM on Human-Computer Interaction, 2022, 6(CSCW2): 1-27.
[78]	JUNG J, KIM B, LEE J Y, et al. Robust upright adjustment of 360 spherical panoramas[J]. The Visual Computer, 2017, 33(6): 737-747. DOI URL
[79]	KOPF J. 360° video stabilization[J]. ACM Transactions on Graphics, 2016, 35(6): 1-9.
[80]	SHEN L C, HUANG T K, CHEN C S, et al. A 2.5D approach to 360 panorama video stabilization[C]// 2018 25th IEEE International Conference on Image Processing. New York: IEEE Press, 2018: 3184-3188.
[81]	TANG C Z, WANG O, LIU F, et al. Joint stabilization and direction of 360° videos[J]. ACM Transactions on Graphics, 2019, 38(2): 1-13.
[82]	CHEN Z Z, LI Y M, ZHANG Y X. Recent advances in omnidirectional video coding for virtual reality: projection and evaluation[J]. Signal Processing, 2018, 146: 66-78. DOI URL
[83]	GAO J H, BROWN M S. An interactive editing tool for correcting panoramas[C]// SA ’12: SIGGRAPH Asia 2012 Technical Briefs. New York: ACM, 2012: 31:1-31:4.
[84]	ZHU Z, MARTIN R R, HU S M. Panorama completion for street views[J]. Computational Visual Media, 2015, 1(1): 49-57. DOI URL
[85]	SHANG Z Y, LIU Y W, LI G Y, et al. Viewport-oriented panoramic image inpainting[C]// 2022 IEEE International Conference on Image Processing. New York: IEEE Press, 2022: 3031-3035.
[86]	XU B B, PATHAK S, FUJII H, et al. Spatio-temporal video completion in spherical image sequences[J]. IEEE Robotics and Automation Letters, 2017, 2(4): 2032-2039. DOI URL
[87]	ZHAO Q, WAN L, FENG W, et al. 360 panorama cloning on sphere[EB/OL]. [2022-12-12]. https://arxiv.org/abs/1709.01638.
[88]	ZHANG Y, ZHANG F L, ZHU Z, et al. Fast edit propagation for 360 degree panoramas using function interpolation[J]. IEEE Access, 2022, 10: 43882-43894. DOI URL
[89]	ZHANG Y P, ZHANG H Z, LI D J, et al. Toward real-world panoramic image enhancement[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2020: 2675-2684.
[90]	LIU H Y, RUAN Z B, FANG C W, et al. A single frame and multi-frame joint network for 360-degree panorama video super-resolution[EB/OL]. [2022-12-01]. https://arxiv.org/abs/2008.10320.
[91]	YOON Y, CHUNG I, WANG L, et al. SphereSR: 360° image super-resolution with arbitrary projection via continuous spherical image representation[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 5667-5676.
[92]	WANG M, LYU X Q, LI Y J, et al. VR content creation and exploration with deep learning: a survey[J]. Computational Visual Media, 2020, 6(1): 3-28. DOI
[93]	ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5967-5976.
[94]	PARK T, LIU M Y, WANG T C, et al. Semantic image synthesis with spatially-adaptive normalization[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 2332-2341.

全景图像视频的场景分析与内容处理方法综述

Survey of methods for scene analysis and content processing in panoramic images and videos

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 18

参考文献 94

相关文章 15

编辑推荐

Metrics

本文评价

[1]	韩兆阳, 翁冬冬, 郭署山, 贺文杰, 江海燕, 李冬. 一种基于简易标记点编码的光学跟踪系统 [J]. 图学学报, 2023, 44(5): 997-1012.
[2]	朱永宁 , 葛婷 , 杜盛瑀 , 楼泽如 , 王建民 . 虚拟现实全景流体绘画系统的可用性研究[J]. 图学学报, 2021, 42(5): 833-840.
[3]	赵建军, 黄竣鹏, 陈俊良. 基于 Leap Motion 的电影前期预演人机交互方法[J]. 图学学报, 2021, 42(1): 71-78.
[4]	郑明钰, 李家和, 张晗, 骆岩林, 申佳丽, 朱小明. 支持力反馈的沉浸式物理学习环境的构建 [J]. 图学学报, 2021, 42(1): 79-86.
[5]	滕健，黄佳慧，宫凯 . 基于 BCI 的虚拟现实模拟驾驶教学系统设计[J]. 图学学报, 2020, 41(2): 217-223.
[6]	邱远航 1，孙贤波 1，刘勇弟 1，蔡正清 1，徐宏勇 2 . 污水处理厂虚拟现实教学软件开发及应用[J]. 图学学报, 2020, 41(2): 233-236.
[7]	林莹莹，蔡睿凡，朱雨真，唐祥峻，金小刚. 基于Leap Motion 的虚拟现实陶艺体验系统[J]. 图学学报, 2020, 41(1): 57-65.
[8]	李祥 1,2，王学文 1,2，谢嘉成 1,2，乔春光 1,2，杨兆建 1,2 . 复杂工况下采运装备虚拟运行关键技术研究[J]. 图学学报, 2019, 40(2): 403-409.
[9]	姜立军1,2，李建余1，李哲林1,2，吴章鸿1，张瑜1. 虚拟骑行中的三种导航界面的体验度量[J]. 图学学报, 2018, 39(3): 515-521.
[10]	冯桂珍1，池建斌1，邢海军1，张增强1，贾双2. 基于Unity3D 的减速器虚拟拆装实验[J]. 图学学报, 2018, 39(2): 304-308.
[11]	孙辉，吕健，寸文哲. VR 系统信息可视化模型[J]. 图学学报, 2018, 39(2): 317-326.
[12]	朱晓强1,2，周浩1,2，宋磊1,2. 基于α-shape 的树冠LOD 建模[J]. 图学学报, 2017, 38(3): 303-313.
[13]	刘伟，杜强，张顺心，张建军. 基于VR 技术的工程图学移动学习系统[J]. 图学学报, 2016, 37(6): 857-.
[14]	王新翔，赵罡，肖文磊. 虚拟飞行场景中事故特效的实时仿真[J]. 图学学报, 2016, 37(2): 243-248.
[15]	王学文，秦毅，杨昕宇，庞新宇，杨兆建. 采煤机虚拟现实装配仿真系统研究[J]. 图学学报, 2015, 36(2): 268-273.