Spatiotemporal data visualization based on density map multi-target tracking

doi:10.11996/JG.j.2095-302X.2024061289

Abstract

Abstract:

The spatiotemporal data tracking visualization has received widespread attention. The focus of this research is on depicting the dynamic details of the data and ensuring trajectory consistency with the observation results. In this paper, a model that combined deep learning with traditional tracking techniques was proposed to perform tracking tasks, thereby improving the speed and accuracy of visualization. First, a high-quality Perlin noise dataset was generated, on which a multi-target tracking model was trained. Second, a two-stage, multi-model deep learning framework was proposed to enhance the analysis depth of dynamic scenes. Finally, in order to continuously display detailed tracking information, a visualization solution that combined trajectories and vector fields was introduced to enhance the visual effect of tracking information. Different cases in this study demonstrated the usefulness and robustness of the proposed method, quantitatively evaluating and omparing the method from multiple aspects. The results showed that the method proposed in this study can help users in understanding multi-target tracking information in different scenarios.

Key words: data visualization, deep learning, spatial-temporal data, multiple-object tracking

CLC Number:

TP391

SONG Sicheng, CHEN Chen, LI Chenhui, WANG Changbo. Spatiotemporal data visualization based on density map multi-target tracking[J]. Journal of Graphics, 2024, 45(6): 1289-1300.

Figures/Tables 14

Fig. 1 Challenges of multi-target tracking visualization ((a) Generalization problem of the model; (b) Missing information about target morphological changes; (c) Target segmentation under different thresholds)

Fig. 2 Overview of the paper structure

Fig. 3 Multi-target tracking model

Fig. 4 Schematic diagram of the training process of PG-Flow

Fig. 5 Visual symbols in target tracking ((a) Positioning mark; (b) Time gradient indication; (c) Micromotion enhancement; (d) Flow field visualization)

Fig. 6 Dynamic visualization of vector fields ((a) Initial noise image; (b) Optical flow visualization; (c) Visualization of all fields; (d) Visualization of key fields)

Fig. 7 Visualization of random multi-target tracking ((a) Input image; (b) Tracking result)

Fig. 8 Crowd tracking visualization for the shopping mall dataset ((a) Shopping mall crowd dataset; (b) Density map conversion result; (c) Crowd tracking visualization result)

Fig. 9 Visualization of CO2 emissions ((a) Visualization of CO2 changes in the Southern Hemisphere on Oct. 5, 2006; (b) Visualization of CO2 changes in the Southern Hemisphere on Oct. 25, 2006)

Fig. 10 Evaluation results of random multi-target tracking with Perlin noise

Fig. 11 Comparison of tracking quantification between this method and traditional methods ((a) Variance of trajectory length; (b) Trajectory length with a length threshold of 10)

Fig. 12 Comparison of tracking trajectories between this method and traditional methods ((a) Tracking results of the traditional method; (b) Tracking results of this method)

Fig. 13 Comparison between FlowNet2, PG-Flow and PG-Flow-1 ((a) Continuous Perlin noise data; (b) Crowd in a shopping mall data)

Fig. 14 Comparison of the similarity between different model tracking results and the ground truth

References 47

[1]	DORAISWAMY H, NATARAJAN V, NANJUNDIAH R S. An exploration framework to identify and track movement of cloud systems[J]. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(12): 2896-2905.
[2]	PARK Y, LEPETIT V, WOO W. Extended keyframe detection with stable tracking for multiple 3D object tracking[J]. IEEE Transactions on Visualization and Computer Graphics, 2011, 17(11): 1728-1735.
[3]	VALSANGKAR A A, MONTEIRO J M, NARAYANAN V, et al. An exploratory framework for cyclone identification and tracking[J]. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(3): 1460-1473.
[4]	REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1134-1149.
[5]	ILG E, MAYER N, SAIKIA T, et al. FlowNet 2.0: evolution of optical flow estimation with deep networks[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1647-1655.
[6]	CABRAL B, LEEDOM L C. Imaging vector fields using line integral convolution[C]// The 20th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1993: 263-270.
[7]	SMEULDERS A W M, CHU D M, CUCCHIARA R, et al. Visual tracking: an experimental survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(7): 1442-1468. DOI PMID
[8]	WANG N Y, YEUNG D Y. Ensemble-based tracking: aggregating crowdsourced structured time series data[C]// The 31st International Conference on International Conference on Machine Learning. Cambridge: JMLR.Org, 2014: II-1107-II-1115.
[9]	OESTERLING P, HEINE C, WEBER G H, et al. Computing and visualizing time-varying merge trees for high-dimensional data[M]//CARR H, GARTH C, WEINKAUF T. Topological Methods in Data Analysis and Visualization: Theory, Algorithms, and Applications. Cham: Springer, 2017: 87-101.
[10]	MUTHUMANICKAM P K, VROTSOU K, NORDMAN A, et al. Identification of temporally varying areas of interest in long-duration eye-tracking data sets[J]. IEEE Transactions on Visualization and Computer Graphics, 2019, 25(1): 87-97.
[11]	RAUTENHAUS M, BÖTTINGER M, SIEMEN S, et al. Visualization in meteorology-a survey of techniques and tools for data analysis tasks[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(12): 3268-3296.
[12]	SBALZARINI I F, KOUMOUTSAKOS P. Feature point tracking and trajectory analysis for video imaging in cell biology[J]. Journal of Structural Biology, 2005, 151(2): 182-195. PMID
[13]	COMANICIU D, RAMESH V, MEER P. Kernel-based object tracking[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(5): 564-577.
[14]	WANG L, TAN T N, NING H Z, et al. Silhouette analysis-based gait recognition for human identification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(12): 1505-1518.
[15]	CIAPARRONE G, SÁNCHEZ F L, TABIK S, et al. Deep learning in video multi-object tracking: a survey[J]. Neurocomputing, 2020, 381: 61-88.
[16]	DANELLJAN M, HÄGER G, KHAN F S, et al. Convolutional features for correlation filter based visual tracking[C]// 2015 IEEE International Conference on Computer Vision Workshop. New York: IEEE Press, 2015: 621-629.
[17]	BEWLEY A, GE Z Y, OTT L, et al. Simple online and realtime tracking[C]// 2016 IEEE International Conference on Image Processing. New York: IEEE Press, 2016: 3464-3468.
[18]	REDMON J, FARHADI A. Yolov3: an incremental improvement[EB/OL]. [2024-05-08]. https://arxiv.org/abs/1804.02767.
[19]	LIN T Y, DOLLÁR P, GIRSHICK R B, et al. Feature pyramid networks for object detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 936-944.
[20]	POPPE R. Vision-based human motion analysis: an overview[J]. Computer Vision and Image Understanding, 2007, 108(1/2): 4-18.
[21]	WANG L, HU W M, TAN T J. Recent developments in human motion analysis[J]. Pattern Recognition, 2003, 36(3): 585-601.
[22]	ZHOU T H, BROWN M, SNAVELY N, et al. Unsupervised learning of depth and ego-motion from video[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 6612-6619.
[23]	SAQIB M, KHAN S D, SHARMA N, et al. Extracting descriptive motion information from crowd scenes[C]// 2017 International Conference on Image and Vision Computing New Zealand. New York: IEEE Press, 2017: 1-6.
[24]	BROX T, MALIK J. Large displacement optical flow: descriptor matching in variational motion estimation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(3): 500-513. DOI PMID
[25]	JOBARD B, RAY N, SOKOLOV D. Visualizing 2D flows with animated arrow plots[EB/OL]. [2024-05-08]. https://arxiv.org/abs/1205.5204.
[26]	LEFER W, JOBARD B, LEDUC C. High-quality animation of 2D steady vector fields[J]. IEEE Transactions on Visualization and Computer Graphics, 2004, 10(1): 2-14. PMID
[27]	CHEN S, LI S H, CHEN S M, et al. R-Map: a map metaphor for visualizing information reposting process in social media[J]. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(1): 1204-1214.
[28]	WU Y C, LIU S X, YAN K, et al. OpinionFlow: visual analysis of opinion diffusion on social media[J]. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(12): 1763-1772.
[29]	LI C H, BACIU G, WANG Y Z, et al. DDLVis: real-time visual query of spatiotemporal data distribution via density dictionary learning[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 28(1): 1062-1072.
[30]	KIM S, JEONG S, WOO I, et al. Data flow analysis and visualization for spatiotemporal statistical data without trajectory information[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(3): 1287-1300.
[31]	LI C H, BACIU G, HAN Y. StreamMap: smooth dynamic visualization of high-density streaming points[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(3): 1381-1393.
[32]	CHEN C, WANG C B, BAI X, et al. GenerativeMap: visualization and exploration of dynamic density maps via generative learning model[J]. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(1): 216-226.
[33]	刘灿, 赖楚凡, 蒋瑞珂, 等. 深度学习驱动的可视化[J]. 计算机辅助设计与图形学学报, 2020, 32(10): 1537-1548.
	LIU C, LAI C F, JIANG R K, et al. Visualization driven by deep learning[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(10): 1537-1548. (in Chinese)
[34]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 770-778.
[35]	SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, inception-ResNet and the impact of residual connections on learning[C]// The 31st AAAI Conference on Artificial Intelligence. Palo Alto: AAAI, 2017: 4278-4284.
[36]	RIBEIRO M I. Kalman and extended Kalman filters: concept, derivation and properties[EB/OL]. [2024-05-08]. https://people.duke.edu/-hpgavin/SystemID/References/Ribeiro-KalmanFilter-2004.pdf.
[37]	LUETTEKE F, ZHANG X, FRANKE J. Implementation of the Hungarian Method for object tracking on a camera monitored transportation system[C]// ROBOTIK 2012; 7th German Conference on Robotics. New York: IEEE Press, 2012: 1-6.
[38]	BOTEV Z, GROTOWSKI J, KROESE D P. Kernel density estimation via diffusion[J]. The Annals of Statistics, 2010, 38(5): 2916-2957.
[39]	PERLIN K. Improving noise[J]. ACM Transactions on Graphics (TOG), 2002, 21(3): 681-682.
[40]	BAKER S, SCHARSTEIN D, LEWIS J, et al. A database and evaluation methodology for optical flow[J]. International Journal of Computer Vision, 2011, 92(1): 1-31.
[41]	FARNEBÄCK G. Two-frame motion estimation based on polynomial expansion[C]// The 13th Scandinavian Conference on Image Analysis. Cham: Springer, 2003: 363-370.
[42]	HERSHBERGER J, SNOEYINK J. Speeding up the douglas-peucker line-simplification algorithm[EB/OL]. [2024-05-08]. https://dl.acm.org/doi/book/10.5555/902273.
[43]	BAO P, ZHANG L, WU X L. Canny edge detection enhancement by scale multiplication[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(9): 1485-1490. PMID
[44]	CHEN K, LOY C C, GONG S, et al. Feature mining for localised crowd counting[EB/OL]. [2024-05-08]. https://personal.ie.cuhk.edu.hk/-ccloy/project_feat_mine_count/index.html.
[45]	VENKATESAN R, KOON S M, JAKUBOWSKI M H, et al. Robust image hashing[C]// 2000 International Conference on Image Processing. New York: IEEE Press, 2000: 664-666.
[46]	VEGA F, MEDINA J, MENDOZA D, et al. A robust video identification framework using perceptual image hashing[C]// 2017 XLIII Latin American Computer Conference. New York: IEEE Press, 2017: 1-10.
[47]	陈辰. 基于空间特征挖掘的时空数据可视化研究[D]. 上海: 华东师范大学, 2020.
	CHEN C. Research on spatio-temporal data visualization based on spatial features mining[D]. Shanghai: East China Normal University, 2020.. (in Chinese)