An optical tracking system based on simple marker encoding

doi:10.11996/JG.j.2095-302X.2023050997

Abstract

Abstract:

In visual, augmented, or mixed reality applications, real-time acquisition of user and object poses is a prerequisite for building a highly immersive virtual environment. With the continuous development of virtual reality technology, users’ demands for the range of motion in virtual environments have been increasing. They are no longer content with limited movement within the confined space of a single room; instead, they seek to roam and interact in a larger range of environments. Most of the tracking systems used by popular AR/VR devices today are designed for room-level or even smaller range tracking. When larger range tracking is required, these systems either introduce greater error drift or require more hardware to be arranged in the room to cover a larger area (e.g. Light House), which creates huge hardware costs and a complex configuration process, making them not suitable for general and personal use. To address this, a system for optical positioning and tracking was proposed, which could achieve accurate 3D camera tracking by deploying a small number of infrared LED markers on the ceiling or floor. The proposed tracking system utilized the most basic dot and line elements to build the landmark pattern. Compared with traditional marker-based systems, individual dots do not contain any information and are identified only after they are formed into a basic graphic element with a line next to them. The straight line segments exist to increase the redundancy of the basic graph elements, thus avoiding the situation where the dots are obscured and cannot be recognized. By designing the encoding principle of the marker patterns, employing the layout repeated feature retrieval method, and implementing the corresponding points matching algorithms, the fast and accurate decoding of the landmark images was realized. Experiments have proven that the system could achieve the position accuracy at the millimeter level. In robustness experiments, the proposed method could maintain higher recognition accuracy even in the presence of challenges such as large inclination angles and marker point occlusion. These measurements show the potential of our system to cope with more extreme situations. We also count the processing time of the system, and the average latency of our method is 4.34 ms, which indicates that performing sparse graph element layout and simplifying marker point decoding effectively reduces the system computation time. The resulting tracking system possesses characteristics such as low cost, easy scalability, and resilience to occlusion, thereby meeting the demand for real-time tracking and positioning at the 100 square meter level.

Key words: optical tracking, marker encoding, image processing, positioning and navigation, VR system

CLC Number:

TP391.9

HAN Zhao-yang, WENG Dong-dong, GUO Shu-shan, HE Wen-jie, JIANG Hai-yan, LI Dong. An optical tracking system based on simple marker encoding[J]. Journal of Graphics, 2023, 44(5): 997-1012.

Figures/Tables 24

Fig. 1 System diagram (a) Tracking objects and tracking devices; (b) Mark point pattern)

Fig. 2 Basic elements ((a) Composition of basic element; (b) Direction of basic element)

Fig. 3 Geometric invariants ((a) Combination of elements with parallel main direction; (b) Combination of elements perpendicular to the main direction)

Fig. 4 Minimum field of view ((a) Parallel elements in the minimum field of view; (b) Vertical elements in the minimum field of view)

Fig. 5 Location to be selected and random distribution ((a) Discrete elements location; (b) Automated layout result)

Fig. 6 Density detection

Fig. 7 Uniqueness detection

Fig. 8 The process of encoding and decoding

Fig. 9 Perspective imaging

Fig. 10 Identification of basic elements ((a) Binary image; (b) Dot detection; (c) Line detection; (d) Element recognition)

Fig. 11 Geometric invariant restoration ((a) Basic element; (b) Direction recognition; (c) Outer elements detection; (d) Reprojection; (e) Scale)

Fig. 12 Virtual environment for experiment

Fig. 13 The three-dimensional position distribution of the camera at an inclination angle of 20° ((a) 3D position; (b) XZ section; (c) YZ section)

Fig. 14 Reprojection errors in different directions ((a) Z=4; (b) Z=6)

Table 1 Average absolute error of three-dimensional position (mm)

误差	X	Y	Z
平均误差	5.693	6.060	10.710
最大误差	25.322	25.218	112.941

Table 2 Average absolute error of three-dimensional orientation (°)

误差	Pitch	Yaw	Roll
平均误差	0.347	0.054	0.133
最大误差	1.923	0.283	1.026

Table 3 Average processing time per frame (ms)

时间	RTP	LWT	SOT	Ours
处理时间	13.00	11.00	5.00	4.34

Fig. 15 Perspective experiment

Fig. 16 Jitter experiment

Fig. 17 Occlusion experiment

Fig. 18 Experimental environment settings

Fig. 19 Three-dimensional trace ((a) YZ direction; (b) XZ direction)

Fig. 20 Reprojection error under different occlusion areas ((a) 20% occlusion area; (b) 70% occlusion area)

Fig. 21 Stable tracking under occlusion

References 34

[1]	SUTHERLAND I E. A head-mounted three dimensional display[C]// Proceedings of the December 9-11, 1968, Fall Joint Computer Conference, Part I. New York: ACM, 1968: 757-764.
[2]	SAYYAD E, SRA M, HÖLLERER T. Walking and teleportation in wide-area virtual reality experiences[C]// 2020 IEEE International Symposium on Mixed and Augmented Reality. New York: IEEE Press, 2020: 608-617.
[3]	BARAI S, MOMIN M. Outside-in electromagnetic tracking method for augmented and virtual reality 6-degree of freedom head-mounted displays[C]// The 4th International Conference on Intelligent Computing and Control Systems. New York: IEEE Press, 2020: 467-476.
[4]	WILLIAMS B, NARASIMHAM G, RUMP B, et al. Exploring large virtual environments with an HMD when physical space is limited[C]// The 4th Symposium on Applied Perception in Graphics and Visualization. New York: ACM, 2007: 41-48.
[5]	HOFMANN-WELLENHOF B, LICHTENEGGER H, COLLINS J. Global positioning system: theory and practice[M]. Vienna: Springer Vienna, 1997: 12-23.
[6]	HARLE R K, HOPPER A. Deploying and evaluating a location-aware system[C]// The 3rd International Conference on Mobile Systems, Applications, and Services. New York: ACM, 2005: 219-232.
[7]	PINTARIC T, KAUFMANN H. Affordable infrared-optical pose-tracking for virtual and augmented reality[C]// Proceedings of Trends and Issues in Tracking for Virtual Environments Workshop. New York: IEEE Press, 2007: 44-51.
[8]	LI D, WANG D L, WENG D D, et al. Coded light based extensible optical tracking system[C]// 2018 IEEE Conference on Virtual Reality and 3D User Interfaces. New York: IEEE Press, 2018: 439-445.
[9]	KLEIN G, MURRAY D. Parallel tracking and mapping for small AR workspaces[C]// The 6th IEEE and ACM International Symposium on Mixed and Augmented Reality. New York: IEEE Press, 2008: 225-234.
[10]	CASTLE R, KLEIN G, MURRAY D W. Video-rate localization in multiple maps for wearable augmented reality[C]// The 12th IEEE International Symposium on Wearable Computers. New York: IEEE Press, 2009: 15-22.
[11]	DAVISON. Real-time simultaneous localisation and mapping with a single camera[C]// The 9th IEEE International Conference on Computer Vision. New York: IEEE Press, 2008: 1403-1410.
[12]	SCARAMUZZA D, FRAUNDORFER F. Visual odometry[tutorial[J]. IEEE Robotics & Automation Magazine, 2011, 18(4): 80-92.
[13]	BISHOP T G. Self-tracker: a smart optical sensor on silicon[M]. Chapel Hill: The University of North Carolina at Chapel Hill, 1984: 1-78
[14]	WANG J F, CHI V, FUCHS H. A real-time optical 3D tracker for head-mounted display systems[C]// Proceedings of the 1990 Symposium on Interactive 3D Graphics. New York: ACM, 1990: 205-215.
[15]	WARD M, AZUMA R, BENNETT R, et al. A demonstrated optical tracker with scalable work area for head-mounted display systems[C]// Proceedings of the 1992 Symposium on Interactive 3D Graphics. New York: ACM, 1992: 43-52.
[16]	WELCH G, BISHOP G, VICCI L, et al. The HiBall Tracker: high-performance wide-area tracking for virtual and augmented environments[C]// Proceedings of the ACM Symposium on Virtual Reality Software and Technology. New York: ACM, 1999: 1-10.
[17]	NAKAZATO Y, KANBARA M, YOKOYA N. Localization of wearable users using invisible retro-reflective markers and an IR camera[C]// Proc SPIE 5664, Stereoscopic Displays and Virtual Reality Systems XII. Bellingham: SPIE, 2005: 563-570.
[18]	KATO H, BILLINGHURST M. Marker tracking and HMD calibration for a video-based augmented reality conferencing system[C]// The 2nd IEEE and ACM International Workshop on Augmented Reality. New York: IEEE Press, 1999: 85-94.
[19]	FIALA M. Artag, a fiducial marker system using digital techniques[J]. National Research Council Publication, 2004, 47419: 1-47.
[20]	FOXLIN E, NAIMARK L. VIS-Tracker: a wearable vision-inertial self-tracker[C]// IEEE Virtual Reality, 2003. Proceedings. New York: IEEE Press, 2003: 199-206.
[21]	MAESEN S, LIU Y J, GOORTS P, et al. Tile Tracker: a practical and inexpensive positioning system for mobile AR applications[C]// International Conference on Augmented and Virtual Reality. Cham: Springer, 2014: 434-441.
[22]	SAGITOV A, SHABALINA K, LAVRENOV R, et al. Comparing fiducial marker systems in the presence of occlusion[C]// 2017 International Conference on Mechanical, System and Control Engineering. New York: IEEE Press, 2017: 377-382.
[23]	赵梓建. 大空间沉浸式虚拟现实系统的视觉定位研究[D]. 秦皇岛: 燕山大学, 2020.
	ZHAO Z J. Research on visual positioning of large space immersive virtual reality system[D]. Qinhuangdao: Yanshan University, 2020. (in Chinese)
[24]	MAESEN S, GOORTS P, BEKAERT P. Scalable optical tracking for navigating large virtual environments using spatially encoded markers[C]// The 19th ACM Symposium on Virtual Reality Software and Technology. New York: ACM, 2013: 101-110.
[25]	JORISSEN L, MAESEN S, DOSHI A, et al. Robust global tracking using a seamless structured pattern of dots[C]// International Conference on Augmented and Virtual Reality. Cham: Springer, 2014: 210-231.
[26]	BERGAMASCO F, ALBARELLI A, TORSELLO A. Pi-Tag: a fast image-space marker design based on projective invariants[J]. Machine Vision and Applications, 2013, 24(6): 1295-1310. DOI URL
[27]	UCHIYAMA H, SAITO H. Random dot markers[C]// 2011 IEEE Virtual Reality Conference. New York: IEEE Press, 2011: 35-38.
[28]	CHEN L, FU H B, ANDY LI W H, et al. Scalable maps of random dots for middle-scale locative mobile games[C]// 2013 IEEE Virtual Reality. New York: IEEE Press, 2013: 39-42.
[29]	NAKAI T, KISE K, IWAMURA M. Use of affine invariants in locally likely arrangement hashing for camera-based document image retrieval[M]// Document Analysis Systems VII. Heidelberg: Springer, 2006: 541-552.
[30]	ZHANG Z. A flexible new technique for camera calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11): 1330-1334. DOI URL
[31]	GAO X S, HOU X R, TANG J L, et al. Complete solution classification for the perspective-three-point problem[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(8): 930-943. DOI URL
[32]	LEPETIT V, MORENO-NOGUER F, FUA P. EPnP: an accurate O(n) solution to the PnP problem[J]. International Journal of Computer Vision, 2009, 81(2): 155-166. DOI URL
[33]	MAESEN S, BEKAERT P. Low-cost, wide-area tracking for virtual environments[C]// IEEE VR 2007 Workshop Trends and Issues in Tracking for Virtual Environments. New York: IEEE Press, 2007: 16-21.
[34]	YANG L M, NORMAND J M, MOREAU G. Local geometric consensus: a general purpose point pattern-based tracking algorithm[J]. IEEE Transactions on Visualization and Computer Graphics, 2015, 21(11): 1299-1308. DOI URL