一种基于简易标记点编码的光学跟踪系统

doi:10.11996/JG.j.2095-302X.2023050997

图学学报 ›› 2023, Vol. 44 ›› Issue (5): 997-1012.DOI: 10.11996/JG.j.2095-302X.2023050997

• 计算机图形学与虚拟现实 • 上一篇下一篇

一种基于简易标记点编码的光学跟踪系统

韩兆阳¹^,²(), 翁冬冬¹^,², 郭署山³, 贺文杰¹^,², 江海燕¹^,², 李冬¹^,²()

1.北京市混合现实与新型显示工程技术研究中心，北京 100081
2.北京理工大学光电学院，北京 100081
3.北京临近空间飞艇技术开发有限公司，北京 100070

收稿日期:2023-02-12 接受日期:2023-06-06 出版日期:2023-10-31 发布日期:2023-10-31
通讯作者: 李冬(1986-)，男，助理研究员，博士。主要研究方向为虚拟现实、增强现实和人机交互。E-mail：drli@bit.edu.cn
作者简介:韩兆阳(1998-)，男，硕士研究生。主要研究方向为虚拟现实、增强现实和人机交互。E-mail：hzy3_3@163.com
基金资助:
国家国防科技工业局基础科研项目(JCKY2019205A004);国家自然科学基金项目(62072036)

An optical tracking system based on simple marker encoding

HAN Zhao-yang¹^,²(), WENG Dong-dong¹^,², GUO Shu-shan³, HE Wen-jie¹^,², JIANG Hai-yan¹^,², LI Dong¹^,²()

1. Beijing Engineering Research Center of Mixed Reality and Advanced Display, Beijing 100081 China
2. School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081 China
3. Beijing Near Space Airship Technology Development Co, Ltd, Beijing 100070 China

Received:2023-02-12 Accepted:2023-06-06 Online:2023-10-31 Published:2023-10-31
Contact: LI Dong (1986-), assistant researcher, Ph.D. His main research interests cover visual reality, augmented reality and human-computer interaction. E-mail：drli@bit.edu.cn
About author:HAN Zhao-yang (1998-), master student. His main research interests cover visual reality, augmented reality and human-computer interaction. E-mail：hzy3_3@163.com
Supported by:
Basic Research Project of State Administration of Science(JCKY2019205A004);National Natural Science Foundation of China(62072036)

摘要/Abstract

摘要：

在虚拟现实，增强现实或是混合现实应用中，实时获取用户和交互对象的位姿是构建高沉浸感虚拟环境的前提条件。随着虚拟现实技术的不断发展，用户对虚拟环境中运动范围的要求不断提高，不再仅满足于单个房间内的小范围移动，而希望能在更大范围进行漫游和交互。为此，提出了一种光学定位追踪系统，通过在天花板或地面布置少量的红外LED标记点来实现精确的相机三维追踪。所述跟踪系统使用了最基本的点、线元素构建标记点图案，通过设计该标记点图案的编码原则、布局重复特征检索方法和对应点匹配算法，实现了标记点图像的快速、精确解析。实验证明系统计算的位置精度可以控制在毫米级别范围内，同时在对抗标记点抖动和遮挡等方面保持了较高的识别准确率。由此实现的跟踪系统具有低成本、易拓展和抗遮挡等特点，可以满足百平方米级别范围的实时追踪定位需求。

关键词: 光学追踪, 标记点编码, 图像处理, 定位导航, 虚拟现实系统

Abstract:

In visual, augmented, or mixed reality applications, real-time acquisition of user and object poses is a prerequisite for building a highly immersive virtual environment. With the continuous development of virtual reality technology, users’ demands for the range of motion in virtual environments have been increasing. They are no longer content with limited movement within the confined space of a single room; instead, they seek to roam and interact in a larger range of environments. Most of the tracking systems used by popular AR/VR devices today are designed for room-level or even smaller range tracking. When larger range tracking is required, these systems either introduce greater error drift or require more hardware to be arranged in the room to cover a larger area (e.g. Light House), which creates huge hardware costs and a complex configuration process, making them not suitable for general and personal use. To address this, a system for optical positioning and tracking was proposed, which could achieve accurate 3D camera tracking by deploying a small number of infrared LED markers on the ceiling or floor. The proposed tracking system utilized the most basic dot and line elements to build the landmark pattern. Compared with traditional marker-based systems, individual dots do not contain any information and are identified only after they are formed into a basic graphic element with a line next to them. The straight line segments exist to increase the redundancy of the basic graph elements, thus avoiding the situation where the dots are obscured and cannot be recognized. By designing the encoding principle of the marker patterns, employing the layout repeated feature retrieval method, and implementing the corresponding points matching algorithms, the fast and accurate decoding of the landmark images was realized. Experiments have proven that the system could achieve the position accuracy at the millimeter level. In robustness experiments, the proposed method could maintain higher recognition accuracy even in the presence of challenges such as large inclination angles and marker point occlusion. These measurements show the potential of our system to cope with more extreme situations. We also count the processing time of the system, and the average latency of our method is 4.34 ms, which indicates that performing sparse graph element layout and simplifying marker point decoding effectively reduces the system computation time. The resulting tracking system possesses characteristics such as low cost, easy scalability, and resilience to occlusion, thereby meeting the demand for real-time tracking and positioning at the 100 square meter level.

Key words: optical tracking, marker encoding, image processing, positioning and navigation, VR system

中图分类号:

TP391.9

韩兆阳, 翁冬冬, 郭署山, 贺文杰, 江海燕, 李冬. 一种基于简易标记点编码的光学跟踪系统[J]. 图学学报, 2023, 44(5): 997-1012.

HAN Zhao-yang, WENG Dong-dong, GUO Shu-shan, HE Wen-jie, JIANG Hai-yan, LI Dong. An optical tracking system based on simple marker encoding[J]. Journal of Graphics, 2023, 44(5): 997-1012.

图/表 24

图1 系统示意图((a)追踪对象与追踪设备；(b)标记图案)

Fig. 1 System diagram (a) Tracking objects and tracking devices; (b) Mark point pattern)

图2 基本图元示意图((a)基本图元的组成；(b)基本图元的方向)

Fig. 2 Basic elements ((a) Composition of basic element; (b) Direction of basic element)

图3 基本图元几何不变量((a)主方向平行的图元组合；(b)主方向垂直的图元组合)

Fig. 3 Geometric invariants ((a) Combination of elements with parallel main direction; (b) Combination of elements perpendicular to the main direction)

图4 最小视场范围((a)最小视场中的平行图元；(b)最小视场中的垂直图元)

Fig. 4 Minimum field of view ((a) Parallel elements in the minimum field of view; (b) Vertical elements in the minimum field of view)

图5 基本图元待选位置与随机分布((a)离散图元位点；(b)自动布局结果)

Fig. 5 Location to be selected and random distribution ((a) Discrete elements location; (b) Automated layout result)

图6 密集度检测

Fig. 6 Density detection

图7 独特性检验

Fig. 7 Uniqueness detection

图8 编解码流程

Fig. 8 The process of encoding and decoding

图9 透视成像关系图

Fig. 9 Perspective imaging

图10 基本图元的识别((a)二值图像；(b)检测圆点；(c)检测直线；(d)识别图元)

Fig. 10 Identification of basic elements ((a) Binary image; (b) Dot detection; (c) Line detection; (d) Element recognition)

图11 图像几何不变量还原流程((a)基本图元；(b)识别方向；(c)检测外侧图元；(d)重投影；(e)缩放)

Fig. 11 Geometric invariant restoration ((a) Basic element; (b) Direction recognition; (c) Outer elements detection; (d) Reprojection; (e) Scale)

图12 实验使用的虚拟环境

Fig. 12 Virtual environment for experiment

图13 倾斜角20°时相机三维位置分布((a)三维位置；(b) XZ截面；(c) YZ截面)

Fig. 13 The three-dimensional position distribution of the camera at an inclination angle of 20° ((a) 3D position; (b) XZ section; (c) YZ section)

图14 不同方向的重投影误差

Fig. 14 Reprojection errors in different directions ((a) Z=4; (b) Z=6)

表1 三维位置的平均绝对误差(mm)

Table 1 Average absolute error of three-dimensional position (mm)

误差	X	Y	Z
平均误差	5.693	6.060	10.710
最大误差	25.322	25.218	112.941

表2 三维朝向的平均绝对误差(°)

Table 2 Average absolute error of three-dimensional orientation (°)

误差	Pitch	Yaw	Roll
平均误差	0.347	0.054	0.133
最大误差	1.923	0.283	1.026

表3 每帧平均处理时间(ms)

Table 3 Average processing time per frame (ms)

时间	RTP	LWT	SOT	Ours
处理时间	13.00	11.00	5.00	4.34

图15 大倾角测试

Fig. 15 Perspective experiment

图16 抖动测试

Fig. 16 Jitter experiment

图17 遮挡测试

Fig. 17 Occlusion experiment

图18 实验环境设置

Fig. 18 Experimental environment settings

图19 三维轨迹图((a) YZ方向；(b) XZ方向)

Fig. 19 Three-dimensional trace ((a) YZ direction; (b) XZ direction)

图20 不同遮挡面积下的重投影误差((a)遮挡面积20%；(b)遮挡面积70%)

Fig. 20 Reprojection error under different occlusion areas ((a) 20% occlusion area; (b) 70% occlusion area)

图21 遮挡情况下依然能稳定追踪

Fig. 21 Stable tracking under occlusion

参考文献 34

[1]	SUTHERLAND I E. A head-mounted three dimensional display[C]// Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I. New York: ACM, 1968: 757-764.
[2]	SAYYAD E, SRA M, HÖLLERER T. Walking and teleportation in wide-area virtual reality experiences[C]// 2020 IEEE International Symposium on Mixed and Augmented Reality. New York: IEEE Press, 2020: 608-617.
[3]	BARAI S, MOMIN M. Outside-in electromagnetic tracking method for augmented and virtual reality 6-degree of freedom head-mounted displays[C]// The 4th International Conference on Intelligent Computing and Control Systems. New York: IEEE Press, 2020: 467-476.
[4]	WILLIAMS B, NARASIMHAM G, RUMP B, et al. Exploring large virtual environments with an HMD when physical space is limited[C]// The 4th Symposium on Applied Perception in Graphics and Visualization. New York: ACM, 2007: 41-48.
[5]	HOFMANN-WELLENHOF B, LICHTENEGGER H, COLLINS J. Global positioning system: theory and practice[M]. Vienna: Springer Vienna, 1997: 12-23.
[6]	HARLE R K, HOPPER A. Deploying and evaluating a location-aware system[C]// The 3rd International Conference on Mobile Systems, Applications, and Services. New York: ACM, 2005: 219-232.
[7]	PINTARIC T, KAUFMANN H. Affordable infrared-optical pose-tracking for virtual and augmented reality[C]// Proceedings of Trends and Issues in Tracking for Virtual Environments Workshop. New York: IEEE Press, 2007: 44-51.
[8]	LI D, WANG D L, WENG D D, et al. Coded light based extensible optical tracking system[C]// 2018 IEEE Conference on Virtual Reality and 3D User Interfaces. New York: IEEE Press, 2018: 439-445.
[9]	KLEIN G, MURRAY D. Parallel tracking and mapping for small AR workspaces[C]// The 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. New York: IEEE Press, 2008: 225-234.
[10]	CASTLE R, KLEIN G, MURRAY D W. Video-rate localization in multiple maps for wearable augmented reality[C]// The 12th IEEE International Symposium on Wearable Computers. New York: IEEE Press, 2009: 15-22.
[11]	DAVISON. Real-time simultaneous localisation and mapping with a single camera[C]// The 9th IEEE International Conference on Computer Vision. New York: IEEE Press, 2008: 1403-1410.
[12]	SCARAMUZZA D, FRAUNDORFER F. Visual odometry[tutorial[J]. IEEE Robotics & Automation Magazine, 2011, 18(4): 80-92.
[13]	BISHOP T G. Self-tracker: a smart optical sensor on silicon[M]. Chapel Hill: The University of North Carolina at Chapel Hill, 1984: 1-78
[14]	WANG J F, CHI V, FUCHS H. A real-time optical 3D tracker for head-mounted display systems[C]// Proceedings of the 1990 Symposium on Interactive 3D Graphics. New York: ACM, 1990: 205-215.
[15]	WARD M, AZUMA R, BENNETT R, et al. A demonstrated optical tracker with scalable work area for head-mounted display systems[C]// Proceedings of the 1992 Symposium on Interactive 3D Graphics. New York: ACM, 1992: 43-52.
[16]	WELCH G, BISHOP G, VICCI L, et al. The HiBall Tracker: high-performance wide-area tracking for virtual and augmented environments[C]// Proceedings of the ACM Symposium on Virtual Reality Software and Technology. New York: ACM, 1999: 1-10.
[17]	NAKAZATO Y, KANBARA M, YOKOYA N. Localization of wearable users using invisible retro-reflective markers and an IR camera[C]// Proc SPIE 5664, Stereoscopic Displays and Virtual Reality Systems XII. Bellingham: SPIE, 2005: 563-570.
[18]	KATO H, BILLINGHURST M. Marker tracking and HMD calibration for a video-based augmented reality conferencing system[C]// The 2nd IEEE and ACM International Workshop on Augmented Reality. New York: IEEE Press, 1999: 85-94.
[19]	FIALA M. Artag, a fiducial marker system using digital techniques[J]. National Research Council Publication, 2004, 47419: 1-47.
[20]	FOXLIN E, NAIMARK L. VIS-Tracker: a wearable vision-inertial self-tracker[C]// IEEE Virtual Reality, 2003. Proceedings. New York: IEEE Press, 2003: 199-206.
[21]	MAESEN S, LIU Y J, GOORTS P, et al. Tile Tracker: a practical and inexpensive positioning system for mobile AR applications[C]// International Conference on Augmented and Virtual Reality. Cham: Springer, 2014: 434-441.
[22]	SAGITOV A, SHABALINA K, LAVRENOV R, et al. Comparing fiducial marker systems in the presence of occlusion[C]// 2017 International Conference on Mechanical, System and Control Engineering. New York: IEEE Press, 2017: 377-382.
[23]	赵梓建. 大空间沉浸式虚拟现实系统的视觉定位研究[D]. 秦皇岛: 燕山大学, 2020.
	ZHAO Z J. Research on visual positioning of large space immersive virtual reality system[D]. Qinhuangdao: Yanshan University, 2020. (in Chinese)
[24]	MAESEN S, GOORTS P, BEKAERT P. Scalable optical tracking for navigating large virtual environments using spatially encoded markers[C]// The 19th ACM Symposium on Virtual Reality Software and Technology. New York: ACM, 2013: 101-110.
[25]	JORISSEN L, MAESEN S, DOSHI A, et al. Robust global tracking using a seamless structured pattern of dots[C]// International Conference on Augmented and Virtual Reality. Cham: Springer, 2014: 210-231.
[26]	BERGAMASCO F, ALBARELLI A, TORSELLO A. Pi-Tag: a fast image-space marker design based on projective invariants[J]. Machine Vision and Applications, 2013, 24(6): 1295-1310. DOI URL
[27]	UCHIYAMA H, SAITO H. Random dot markers[C]// 2011 IEEE Virtual Reality Conference. New York: IEEE Press, 2011: 35-38.
[28]	CHEN L, FU H B, ANDY LI W H, et al. Scalable maps of random dots for middle-scale locative mobile games[C]// 2013 IEEE Virtual Reality. New York: IEEE Press, 2013: 39-42.
[29]	NAKAI T, KISE K, IWAMURA M. Use of affine invariants in locally likely arrangement hashing for camera-based document image retrieval[M]// Document Analysis Systems VII. Heidelberg: Springer, 2006: 541-552.
[30]	ZHANG Z. A flexible new technique for camera calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11): 1330-1334. DOI URL
[31]	GAO X S, HOU X R, TANG J L, et al. Complete solution classification for the perspective-three-point problem[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(8): 930-943. DOI URL
[32]	LEPETIT V, MORENO-NOGUER F, FUA P. EPnP: an accurate O(n) solution to the PnP problem[J]. International Journal of Computer Vision, 2009, 81(2): 155-166. DOI URL
[33]	MAESEN S, BEKAERT P. Low-cost, wide-area tracking for virtual environments[C]// IEEE VR 2007 Workshop Trends and Issues in Tracking for Virtual Environments. New York: IEEE Press, 2007: 16-21.
[34]	YANG L M, NORMAND J M, MOREAU G. Local geometric consensus: a general purpose point pattern-based tracking algorithm[J]. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(11): 1299-1308. DOI URL

一种基于简易标记点编码的光学跟踪系统

An optical tracking system based on simple marker encoding

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 24

参考文献 34

相关文章 15

编辑推荐

Metrics

本文评价

[1]	东辉, 陈鑫凯, 孙浩, 姚立纲. 基于改进 YOLOv4 和图像处理的蔬菜田杂草检测[J]. 图学学报, 2022, 43(4): 559-569.
[2]	林森 , 刘旭 . 门控融合对抗网络的水下图像增强 [J]. 图学学报, 2021, 42(6): 948-956.
[3]	满开亮, 汪友生, 刘继荣. 基于稠密残差网络的图像超分辨率重建算法[J]. 图学学报, 2021, 42(4): 556-562.
[4]	王道累, 张天宇. 图像去雾算法的综述及分析[J]. 图学学报, 2020, 41(6): 861-870.
[5]	崔文超, 邹俊杰, 汪方毅, 唐庭龙, 夏平. OBE 理念下项目驱动的数字图像处理教学研究 [J]. 图学学报, 2020, 41(6): 1031-1038.
[6]	吴泽斌 1,张东亮 1,李基拓 2,麻菁 1,信玉峰 3. 复杂场景下的人体轮廓提取及尺寸测量[J]. 图学学报, 2020, 41(5): 740-749.
[7]	李桂，李腾. 基于姿态引导的场景保留人物视频生成[J]. 图学学报, 2020, 41(4): 539-547.
[8]	柳有权 1，张彩荣 1，马雷 2,3，石剑 4，孙昭 1，陈彦云 3 . 一种基于图像结构特征的实时点画生成算法[J]. 图学学报, 2019, 40(3): 435-440.
[9]	张新春，曹应平，韩春雨，白云灿. 基于图像处理的输电线路导线表面损伤特征研究[J]. 图学学报, 2018, 39(3): 440-447.
[10]	李振雨，王好臣. 基于视觉识别定位的苹果采摘系统研究[J]. 图学学报, 2018, 39(3): 493-500.
[11]	李灏，王宏涛，董晴晴. 管道缺陷自动检测与分类[J]. 图学学报, 2017, 38(6): 851-856.
[12]	赵军，田海韬. 利用机器视觉检测马铃薯外部品质方法综述[J]. 图学学报, 2017, 38(3): 382-387.
[13]	高满屯，李阳，王淑侠，王守霞. 一类三焦点曲线[J]. 图学学报, 2016, 37(4): 457-466.
[14]	邹建成，张文婷. 一种基于MOD 字典学习的图像超分辨率重建新算法[J]. 图学学报, 2015, 36(3): 402-406.
[15]	周远，韩裕生，周浦城. 一种单幅图像雨滴去除的方法[J]. 图学学报, 2015, 36(3): 438-443.