Large language model powered UI evaluation system

doi:10.11996/JG.j.2095-302X.2024061178

Abstract

Abstract:

The quality of user interface (UI) design directly impacts product usability and user experience. Designers often face challenges related to consistency and accessibility during the UI design process, increasing cognitive load for users reducing efficiency. Despite awareness of these issues, they currently lack comprehensive knowledge and tools or automatic identification and resolution. To address this challenge, a comprehensive set of UI design evaluation criteria was proposed, covering five key aspects: color, text, layout, control, and icon, specifically targeting consistency and accessibility issues in UI design. Based on these evaluation criteria, a prompt template for evaluating UI consistency and accessibility was developed to enhance the accuracy of large language models (LLMs) like GPT-4 in UI evaluation tasks. Furthermore, a UI evaluation system based on the GPT-4 model was developed. This [26] deeply understood UI design content, automatically detected UI design issues according to the evaluation criteria, and provided targeted improvement suggestions to help designers optimize their UI designs. Experimental results demonstrated that using the prompt template significantly improved the accuracy of GPT-4 in UI evaluations. User studies indicated that employing this UI evaluation system in design practice can significantly enhance the quality of UI designs, thereby boosting product usability and user experience. This system provided designers with an automated UI evaluation tool, offering a new approach to enhancing UI design quality.

Key words: graphical user interface, large language model, UI evaluation, consistency, accessibility

CLC Number:

TP391
TP18

CHEN Xiaojiao, SHU Yunfeng, WANG Ruihan, ZHOU Jiahuan, CHEN Wei. Large language model powered UI evaluation system[J]. Journal of Graphics, 2024, 45(6): 1178-1187.

Figures/Tables 10

Fig. 1 The framework of system

Fig. 2 UI screenshot and UI JSON (partial)

Fig. 3 Dataflow and prompt example

Fig. 4 UI evaluation workflow diagram ((a) UI design prototype view; (b) UI design evaluation feedback view)

Fig. 5 Two UI case studies analysis

Table 1 Comparison of GPT-4 UI evaluation results under different input conditions

输入条件	所有平均设计问题数量	有效平均设计问题数量
仅JSON数据输入	6.3	2.9
仅UI图像数据输入	6.5	1.9
JSON数据+UI图像结合输入	6.6	3.8

Table 2 Comparison of performance indicators for GPT-4 UI evaluation under different input conditions

输入条件	Precision	Recall	F1
仅JSON数据输入	0.472	0.525	0.492
仅UI图像输入	0.316	0.354	0.328
JSON数据+UI图像结合输入	0.587	0.683	0.623

Table 3 Comparison of GPT-4 UI evaluation results under different prompt conditions

提示条件	所有设计问题数量	有效设计问题数量
无提示词	65	37
有提示词	82	62

Table 4 Comparison of UI Design Evaluation Performance Indicators between GPT-4 and Human Evaluators

评估者	Precision	Recall	F1
GPT-4	0.614	0.362	0.455
人类评估师(平均值)	0.802	0.322	0.459

Fig. 6 User study experiment results

References 26

[1]	GRUDIN J. The case against user interface consistency[J]. Communications of the ACM, 1989, 32(10): 1164-1173.
[2]	ZAINA L A M, FORTES R P M, CASADEI V, et al. Preventing accessibility barriers: guidelines for using user interface design patterns in mobile applications[J]. Journal of Systems and Software, 2022, 186: 111213.
[3]	NIELSEN J. Finding usability problems through heuristic evaluation[C]// The SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 1992: 373-380.
[4]	KARAT C M, CAMPBELL R, FIEGEL T. Comparison of empirical testing and walkthrough methods in user interface evaluation[C]// The SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 1992: 397-404.
[5]	段艳花, 刘子建, 宁铎. 基于眼动技术的TMA界面评估及优化[J]. 图学学报, 2022, 43(4): 745-752.
	DUAN Y H, LIU Z J, NING D. Evaluation and optimization of TMA interface supported eye movement technology[J]. Journal of Graphics, 2022, 43(4): 745-752. (in Chinese)
[6]	郑瑞凌, 张俊松. 脑电时空多特征融合的数字图形界面认知负荷评价方法[J]. 计算机辅助设计与图形学学报, 2020, 32(7): 1062-1069.
	ZHENG R L, ZHANG J S. Assessing cognitive load combining features of time, frequency and spatial domain under digital graphical interface[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(7): 1062-1069. (in Chinese)
[7]	PARK S, KIM W, KIM Y H, et al. Computational approaches for app-to-app retrieval and design consistency check[EB/OL]. [2024-03-02]. https://arxiv.org/abs/2309.10328.
[8]	BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]// The 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2020: 159.
[9]	WANG B, LI G, LI Y. Enabling conversational interaction with mobile UI using large language models[C]// 2023 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2023: 432.
[10]	DUAN P T, WARNER J, LI Y, et al. Generating automatic feedback on UI mockups with large language models[C]// 2024 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2024: 6.
[11]	ACHIAM J, ADLER S, AGARWAL S, et al. Gpt-4 technical report[EB/OL]. [2024-03-02]. https://arxiv.org/abs/2303.08774.
[12]	JEFFRIES R, MILLER J R, WHARTON C, et al. User interface evaluation in the real world: a comparison of four techniques[C]// The SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 1991: 119-124.
[13]	ALABOOD L, AMINOLROAYA Z, YIM D, et al. A systematic literature review of the design critique method[J]. Information and Software Technology, 2023, 153: 107081.
[14]	厉向东, 夏涵飞, 单逸飞, 等. 图形用户界面自动生成技术的现状与挑战[J]. 图学学报, 2024, 45(3): 409-421. DOI
	LI X D, XIA H F, SHAN Y F, et al. Opportunities and challenges: automatic generation technologies for graphical user interfaces[J]. Journal of Graphics, 2024, 45(3): 409-421. (in Chinese) DOI
[15]	YANG B, XING Z C, XIA X, et al. Don’t do that! hunting down visual design smells in complex UIs against design guidelines[C]// 2021 IEEE/ACM 43rd International Conference on Software Engineering. New York: IEEE Press, 2021: 761-772.
[16]	LIU Z, CHEN C Y, WANG J J, et al. Nighthawk: fully automated localizing UI display issues via visual understanding[J]. IEEE Transactions on Software Engineering, 2023, 49(1): 403-418.
[17]	YANG B, LI S P. UIGuider: detecting implicit design guidelines using a domain knowledge graph approach[J]. Electronics, 2024, 13(7): 1210.
[18]	GAO S Q, WANG Y H, LIU H X. UiAnalyzer: evaluating whether the UI of apps is at the risk of violating the design conventions in terms of function layout[J]. Expert Systems with Applications, 2024, 239: 122408.
[19]	LU Y W, KNEAREM T, DUTTA S, et al. AI is not enough: a hybrid technical approach to AI adoption in UI linting with heuristics[C]// Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. New York: ACM, 2024: 501.
[20]	XIANG W, ZHU H F, LOU S Q, et al. SimUser: generating usability feedback by simulating various users interacting with mobile applications[C]// 2024 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2024: 9.
[21]	MAHAJAN R, SHNEIDERMAN B. Visual and textual consistency checking tools for graphical user interfaces[J]. IEEE Transactions on Software Engineering, 1997, 23(11): 722-735.
[22]	CHEN X J, TANG X T, ZHAO Y, et al. Evaluating visual consistency of icon usage in across-devices[J]. International Journal of Human-Computer Interaction, 2024, 40(9): 2415-2431.
[23]	刘哲, 王俊杰, 陈春阳, 等. 移动应用程序中用户界面显示缺陷检测的经验研究[EB/OL]. (2024-01-17) [2024-03-29]. http://www.jos.org.cn/1000-9825/7043.htm.
	LIU Z, WANG J J, CHEN C Y, et al. Empirical study on UI display issue detection in mobile applications[EB/OL]. (2024-01-17) [2024-03-29]. http://www.jos.org.cn/1000-9825/7043.htm. (in Chinese)
[24]	KIRKPATRICK A, CONNOR J O, CAMPBELL A, et al. Web content accessibility guidelines (WCAG) 2.1[EB/OL]. (2023-09-21) [2024-02-22]. https://www.w3.org/TR/WCAG21/.
[25]	YAO S Y, ZHAO J, YU D, et al. ReAct: synergizing reasoning and acting in language models[EB/OL]. [2024-02-02]. https://openreview.net/forum?id=WE_vluYUL-X.
[26]	COLLINS C, ANDRIENKO N, SCHRECK T, et al. Guidance in the human-machine analytics process[J]. Visual Informatics, 2018, 2(3): 166-180.