Journal of Graphics ›› 2026, Vol. 47 ›› Issue (4): 766-775.DOI: 10.11996/JG.j.2095-302X.2026040766
• Computer Graphics and Virtual Reality • Previous Articles Next Articles
TANG Xiaoteng1,2, YAO Jun1(
), HU Hefan3, SHAO Jiang1, SHU Yunfeng2
Received:2025-12-23
Accepted:2026-04-28
Online:2026-08-31
Published:2026-08-31
Contact:
YAO Jun
Supported by:CLC Number:
TANG Xiaoteng, YAO Jun, HU Hefan, SHAO Jiang, SHU Yunfeng. User intention recognition for multi-type gaze-based target selection tasks in virtual reality[J]. Journal of Graphics, 2026, 47(4): 766-775.
Add to citation manager EndNote|Ris|BibTeX
URL: http://www.txxb.com.cn/EN/10.11996/JG.j.2095-302X.2026040766
Fig. 3 Experimental stimuli for each task ((a) Icon-triggered task; (b) Text-triggered task; (c) Prompt word-triggered task; (d) Image-triggered task)
| 任务 | 正类(准确率/%) | 负类(准确率/%) |
|---|---|---|
| 1 | 574(83.92) | 110(16.08) |
| 2 | 398(82.23) | 86(17.77) |
| 3 | 479(64.38) | 265(35.62) |
| 4 | 549(80.15) | 136(19.85) |
| 5 | 0 | 754(100) |
Table 1 Statistics of the eye-tracking dataset
| 任务 | 正类(准确率/%) | 负类(准确率/%) |
|---|---|---|
| 1 | 574(83.92) | 110(16.08) |
| 2 | 398(82.23) | 86(17.77) |
| 3 | 479(64.38) | 265(35.62) |
| 4 | 549(80.15) | 136(19.85) |
| 5 | 0 | 754(100) |
| 类型 | 特征 | 数量 |
|---|---|---|
| 基础眼动 | 凝视时间、凝视密度 | 2 |
| 瞳孔直径 | 统计特征 (正值、负值、绝对值、总体值) | 20 |
| 趋势特征 | 9 | |
| 变化率特征 | 9 | |
| 凝视坐标 | 统计特征(X,Y坐标) | 40 |
| 趋势特征(X,Y坐标) | 142 | |
| 变化率特征(X,Y坐标) | 142 |
Table 2 Summary of feature dimensions extracted from eye-tracking data
| 类型 | 特征 | 数量 |
|---|---|---|
| 基础眼动 | 凝视时间、凝视密度 | 2 |
| 瞳孔直径 | 统计特征 (正值、负值、绝对值、总体值) | 20 |
| 趋势特征 | 9 | |
| 变化率特征 | 9 | |
| 凝视坐标 | 统计特征(X,Y坐标) | 40 |
| 趋势特征(X,Y坐标) | 142 | |
| 变化率特征(X,Y坐标) | 142 |
| 模型 | AUC | Accuracy | Precision | Recall | F1-score |
|---|---|---|---|---|---|
| SVM | 0.938 ± 0.012 | 0.873 ± 0.016 | 0.870 ± 0.013 | 0.917 ± 0.021 | 0.893 ± 0.014 |
| LR | 0.924 ± 0.016 | 0.864 ± 0.016 | 0.865 ± 0.025 | 0.907 ± 0.015 | 0.885 ± 0.012 |
| RF | 0.951 ± 0.007 | 0.893 ± 0.010 | 0.880 ± 0.012 | 0.944 ± 0.005 | 0.911 ± 0.008 |
| LightGBM | 0.958 ± 0.008 | 0.900 ± 0.021 | 0.897 ± 0.040 | 0.937 ± 0.021 | 0.916 ± 0.016 |
| MLP | 0.928 ± 0.009 | 0.890 ± 0.005 | 0.890 ± 0.011 | 0.924 ± 0.011 | 0.907 ± 0.005 |
| Bi-LSTM | 0.946 ± 0.010 | 0.872 ± 0.010 | 0.914 ± 0.024 | 0.860 ± 0.013 | 0.886 ± 0.010 |
| 1D CNN | 0.960 ± 0.009 | 0.912 ± 0.010 | 0.921 ± 0.016 | 0.928 ± 0.014 | 0.924 ± 0.010 |
| CNN-LSTM | 0.961 ± 0.008 | 0.912 ± 0.009 | 0.925 ± 0.015 | 0.922 ± 0.017 | 0.923 ± 0.009 |
Table 3 Performance comparison of eight models using five-fold cross-validation
| 模型 | AUC | Accuracy | Precision | Recall | F1-score |
|---|---|---|---|---|---|
| SVM | 0.938 ± 0.012 | 0.873 ± 0.016 | 0.870 ± 0.013 | 0.917 ± 0.021 | 0.893 ± 0.014 |
| LR | 0.924 ± 0.016 | 0.864 ± 0.016 | 0.865 ± 0.025 | 0.907 ± 0.015 | 0.885 ± 0.012 |
| RF | 0.951 ± 0.007 | 0.893 ± 0.010 | 0.880 ± 0.012 | 0.944 ± 0.005 | 0.911 ± 0.008 |
| LightGBM | 0.958 ± 0.008 | 0.900 ± 0.021 | 0.897 ± 0.040 | 0.937 ± 0.021 | 0.916 ± 0.016 |
| MLP | 0.928 ± 0.009 | 0.890 ± 0.005 | 0.890 ± 0.011 | 0.924 ± 0.011 | 0.907 ± 0.005 |
| Bi-LSTM | 0.946 ± 0.010 | 0.872 ± 0.010 | 0.914 ± 0.024 | 0.860 ± 0.013 | 0.886 ± 0.010 |
| 1D CNN | 0.960 ± 0.009 | 0.912 ± 0.010 | 0.921 ± 0.016 | 0.928 ± 0.014 | 0.924 ± 0.010 |
| CNN-LSTM | 0.961 ± 0.008 | 0.912 ± 0.009 | 0.925 ± 0.015 | 0.922 ± 0.017 | 0.923 ± 0.009 |
| Model | AUC | Accuracy | Precision | Recall | F1-score | 模型大小/MB |
|---|---|---|---|---|---|---|
| SVM | 0.951 | 0.890 | 0.880 | 0.939 | 0.908 | 3.01 |
| LR | 0.947 | 0.885 | 0.889 | 0.915 | 0.902 | 0.02 |
| RF | 0.963 | 0.911 | 0.885 | 0.971 | 0.926 | 1.25 |
| LightGBM | 0.970 | 0.944 | 0.935 | 0.971 | 0.953 | 0.35 |
| MLP | 0.935 | 0.892 | 0.904 | 0.909 | 0.907 | 4.34 |
| Bi-LSTM | 0.951 | 0.877 | 0.916 | 0.866 | 0.890 | 0.37 |
| 1D CNN | 0.967 | 0.909 | 0.919 | 0.924 | 0.921 | 3.08 |
| CNN-LSTM | 0.968 | 0.924 | 0.926 | 0.933 | 0.934 | 2.86 |
Table 4 Test-set performance comparison of the eight models
| Model | AUC | Accuracy | Precision | Recall | F1-score | 模型大小/MB |
|---|---|---|---|---|---|---|
| SVM | 0.951 | 0.890 | 0.880 | 0.939 | 0.908 | 3.01 |
| LR | 0.947 | 0.885 | 0.889 | 0.915 | 0.902 | 0.02 |
| RF | 0.963 | 0.911 | 0.885 | 0.971 | 0.926 | 1.25 |
| LightGBM | 0.970 | 0.944 | 0.935 | 0.971 | 0.953 | 0.35 |
| MLP | 0.935 | 0.892 | 0.904 | 0.909 | 0.907 | 4.34 |
| Bi-LSTM | 0.951 | 0.877 | 0.916 | 0.866 | 0.890 | 0.37 |
| 1D CNN | 0.967 | 0.909 | 0.919 | 0.924 | 0.921 | 3.08 |
| CNN-LSTM | 0.968 | 0.924 | 0.926 | 0.933 | 0.934 | 2.86 |
| 模型 | 单个样本平均推理时间/ms | 模型大小/MB | F1-score (测试集) |
|---|---|---|---|
| LightGBM | 42.04 | 0.35 | 0.953 |
| 1D CNN | 154.12 | 3.08 | 0.921 |
| CNN-LSTM | 145.15 | 2.86 | 0.934 |
Table 5 Deployment performance comparison of the three models
| 模型 | 单个样本平均推理时间/ms | 模型大小/MB | F1-score (测试集) |
|---|---|---|---|
| LightGBM | 42.04 | 0.35 | 0.953 |
| 1D CNN | 154.12 | 3.08 | 0.921 |
| CNN-LSTM | 145.15 | 2.86 | 0.934 |
| [1] | BAO Y W, WANG J X, WANG Z M, et al. Exploring 3D interaction with gaze guidance in augmented reality[C]// 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR). New York: IEEE Press, 2023: 22-32. |
| [2] |
HUANG L D, WESTIN T, ELADHARI M P, et al. Eyes can draw: a high-fidelity free-eye drawing method with unimodal gaze control[J]. International Journal of Human-Computer Studies, 2023, 170: 102966.
DOI URL |
| [3] |
GARDONY A L, OKANO K, HUGHES G I, et al. Characterizing information access needs in gaze-adaptive augmented reality interfaces: implications for fast-paced and dynamic usage contexts[J]. Human-Computer Interaction, 2024, 39(5/6): 553-583.
DOI URL |
| [4] | MENGES R, KUMAR C, SENGUPTA K, et al. eyeGUI: a novel framework for eye-controlled user interfaces[C]// The 9th Nordic Conference on Human-Computer Interaction. New York: ACM, 2016: 121. |
| [5] | 杨晓楠, 王帅, 牛红伟, 等. 眼动交互关键技术研究现状与展望[J]. 计算机集成制造系统, 2024, 30(5): 1595-1609. |
| YANG X N, WANG S, NIU H W, et al. Research status and prospects of key technologies of eye-based interaction[J]. Computer Integrated Manufacturing Systems, 2024, 30(5): 1595-1609 (in Chinese). | |
| [6] |
WANG Y Y, TIAN J Z, XIAO L, et al. Research on a spatial-temporal characterisation of blink-triggered eye control interactions[J]. Advanced Engineering Informatics, 2024, 59: 102297.
DOI URL |
| [7] | ADEBAYO S, MCLOONE S, DESSING J C. Hand-eye-object tracking for human intention inference[J]. IFAC-PapersOnLine, 2022, 55(15): 174-179. |
| [8] |
JANG Y M, MALLIPEDDI R, LEE S, et al. Human intention recognition based on eyeball movement pattern and pupil size variation[J]. Neurocomputing, 2014, 128: 421-432.
DOI URL |
| [9] |
CHEN X J, TANG X T, CHEN Y H, et al. Enhancing fixation and pursuit: optimizing field of view and number of targets for selection performance in virtual reality[J]. International Journal of Human-Computer Interaction, 2025, 41(2): 1221-1233.
DOI URL |
| [10] |
TANG X T, CHEN X J, LENG H X, et al. Comparison and optimization of target-assisted gaze input technique for enhanced selection in virtual eye-controlled systems[J]. International Journal of Human-Computer Interaction, 2026, 42(1): 239-257.
DOI URL |
| [11] |
NIU Y F, LI X, YANG W J, et al. Smooth pursuit study on an eye-control system for continuous variable adjustment tasks[J]. International Journal of Human-Computer Interaction, 2023, 39(1): 23-33.
DOI URL |
| [12] |
梁永强, 王崴, 瞿珏, 等. 基于眼动特征的人机交互行为意图预测模型[J]. 电子学报, 2018, 46(12): 2993-3001.
DOI |
| LIANG Y Q, WANG W, QU J, et al. Human-computer interaction behavior and intention prediction model based on eye movement characteristics[J]. Acta electronica sinica, 2018, 46(12): 2993-3001 (in Chinese). | |
| [13] | 侯冠华, 周蓥滢. 交互模态与任务类型对用户交互绩效的影响[J]. 计算机辅助设计与图形学学报, 2025, 37(2): 330-339. |
| HOU G H, ZHOU Y Y. The effect of interaction modality and task type on user interaction efficiency[J]. Journal of Computer-Aided Design & Computer Graphics, 2025, 37(2): 330-339 (in Chinese). | |
| [14] | JANG Y M, LEE S, MALLIPEDDI R, et al. Recognition of human’s implicit intention based on an eyeball movement pattern analysis[C]// The 18th International Conference on Neural Information Processing. Cham: Springer, 2011: 138-145. |
| [15] |
KRAUSZ N E, LAMOTTE D, BATZIANOULIS I, et al. Intent prediction based on biomechanical coordination of EMG and vision-filtered gaze for end-point control of an arm prosthesis[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2020, 28(6): 1471-1480.
DOI PMID |
| [16] |
QU J, GUO H, WANG W, et al. Prediction of human-computer interaction intention based on eye movement and electroencephalograph characteristics[J]. Frontiers in Psychology, 2022, 13: 816127.
DOI URL |
| [17] |
AWAIS M, SAEED M Y, MALIK M S A, et al. Intention based comparative analysis of human-robot interaction[J]. IEEE Access, 2020, 8: 205821-205835.
DOI URL |
| [18] | 胡志明, 李胜, 盖孟. 用户任务预测研究进展与算法分析[J]. 图学学报, 2021, 42(3): 367-375. |
| HU Z M, LI S, GAI M. Research progress of user task prediction and algorithm analysis[J]. Journal of Graphics, 2021, 42(3): 367-375 (in Chinese). | |
| [19] |
PAN Y X, XU J. Gaze-based human intention prediction in the hybrid foraging search task[J]. Neurocomputing, 2024, 587: 127648.
DOI URL |
| [20] | ISOMOTO T, YAMANAKA S, SHIZUKI B. Dwell selection with ML-based intent prediction using only gaze data[J]. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022, 6(3): 120. |
| [21] |
KOOCHAKI F, NAJAFIZADEH L. A data-driven framework for intention prediction via eye movement with applications to assistive systems[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2021, 29: 974-984.
DOI URL |
| [22] | REDDY G S R, PROULX M J, HIRSHFIELD L, et al. Towards an eye-brain-computer interface: combining gaze with the stimulus-preceding negativity for target selections in XR[C]// 2024 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2024: 376. |
| [23] | MINIOTAS D. Application of Fitts’ law to eye gaze interaction[C]// CHI ’00 Extended Abstracts on Human Factors in Computing Systems. New York: ACM, 2000: 339-340. |
| [24] |
FERNANDES A S, MURDISON T S, PROULX M J. Leveling the playing field: a comparative reevaluation of unmodified eye tracking as an input and interaction modality for VR[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(5): 2269-2279.
DOI URL |
| [25] |
YANG S, KIM J, JANG J, et al. Improving probability-based prompt selection through unified evaluation and analysis[J]. Transactions of the Association for Computational Linguistics, 2024, 12: 664-680.
DOI URL |
| [1] | WEN Ruiqi, LV Jian, SONG Dingan, SU Le, LIANG Zhibin. Embodied virtual reality system design and evaluation for rehabilitation training [J]. Journal of Graphics, 2026, 47(4): 776-787. |
| [2] | JI Hailin, ZHANG Yiran, LI Yihang, ZHANG Hongwen, LUO Yanhong. Semi-physical interaction technology for immersive physics experiments [J]. Journal of Graphics, 2026, 47(3): 598-606. |
| [3] | ZHANG Yufei, DING Ding, LI Zhuying. A virtual reality experience for artistic re-creation based on emotion capture [J]. Journal of Graphics, 2025, 46(4): 775-782. |
| [4] | DU Xin, REN Yangfu, XU Senzhe, WANG Juhong, ZHENG Yufei, ZHANG Songhai. Investigation of scene scaling gains in redirected walking within virtual reality [J]. Journal of Graphics, 2025, 46(4): 793-806. |
| [5] | ZHU Xiaoqiang, YANG Yifei. Free sculpting system in virtual reality environment [J]. Journal of Graphics, 2025, 46(2): 345-357. |
| [6] | ZHANG Ting, LAI Jiandu, HOU Guanhua, ZHANG Jingjing. The effects of eye-control speed and target travel distance on user interaction efficiency [J]. Journal of Graphics, 2025, 46(1): 233-240. |
| [7] | YANG Haozhong, KONG Xiaoyu, GU Ruikun, WANG Miao. Research progress and trends in large model technologies for virtual reality [J]. Journal of Graphics, 2024, 45(6): 1117-1131. |
| [8] | LUAN Shuai, WU Jian, FAN Runze, WANG Lili. Observation quality field based collaborative object manipulation in VR [J]. Journal of Graphics, 2024, 45(6): 1338-1348. |
| [9] | REN Yangfu, YU Ge, FU Yueyao, XU Senzhe, HE Yu, WANG Juhong, ZHANG Songhai. The impact of scenery and time on spatial orientation cognition in virtual reality [J]. Journal of Graphics, 2024, 45(6): 1349-1363. |
| [10] | YAN Jiahao, LV Jian, HOU Yukang, MO Xinzhu. Research on the influence of eye movement interaction frequency on visual fatigue in virtual reality [J]. Journal of Graphics, 2024, 45(3): 528-538. |
| [11] | WANG Haomiao, SANG Shengju, DUAN Xiaodong, ZHANG Weihua, TAO Tiwei, MA Ting. Collaborative 3D modeling technique in virtual reality [J]. Journal of Graphics, 2024, 45(1): 169-182. |
| [12] | XIE Hong-xia, HU Yu-ning, ZHANG Yun, WANG Ya-qi, DU Hui, QIN Ai-hong. Survey of methods for scene analysis and content processing in panoramic images and videos [J]. Journal of Graphics, 2023, 44(4): 640-657. |
| [13] | TANG Peng, SA Guo-dong, LIU Zhen-yu, TAN Jian-rong. Design of digital twin system for forging hydraulic press [J]. Journal of Graphics, 2023, 44(3): 609-615. |
| [14] | ZHU Yong-ning , GE Ting , DU Sheng-yu , LOU Ze-ru , WANG Jian-min. A usability study of panoramic fluid painting system in immersive virtual environments [J]. Journal of Graphics, 2021, 42(5): 833-840. |
| [15] | ZHAO Jian-jun, HUANG Jun-peng, CHEN Jun-liang . Human-computer interaction method in previz based on Leap Motion [J]. Journal of Graphics, 2021, 42(1): 71-78. |
| Viewed | ||||||
|
Full text |
|
|||||
|
Abstract |
|
|||||