大语言模型驱动的UI评估系统

doi:10.11996/JG.j.2095-302X.2024061178

图学学报 ›› 2024, Vol. 45 ›› Issue (6): 1178-1187.DOI: 10.11996/JG.j.2095-302X.2024061178

• “大模型与图学技术及应用”专题 • 上一篇下一篇

大语言模型驱动的UI评估系统

陈晓皎¹^,²(), 束云峰¹^,², 汪睿涵¹^,², 周佳欢¹^,², 陈为¹^,²

1.浙江大学计算机辅助设计与图形系统全国重点实验室，浙江杭州 310058
2.浙江大学艺术与考古图像数据实验室，浙江杭州 310058

收稿日期:2024-08-02 接受日期:2024-10-06 出版日期:2024-12-31 发布日期:2024-12-24
第一作者:陈晓皎(1988-)，女，研究员，博士。主要研究方向为人机交互界面、用户体验设计实践、数字人文等。E-mail：chenxiaojiao@zju.edu.cn
基金资助:
国家自然科学基金(52205290);航空基金(2022Z005076001)

Large language model powered UI evaluation system

CHEN Xiaojiao¹^,²(), SHU Yunfeng¹^,², WANG Ruihan¹^,², ZHOU Jiahuan¹^,², CHEN Wei¹^,²

1. State Key Laboratory of CAD&CG, Zhejiang University, Hangzhou Zhejiang 310058, China
2. Laboratory of Art and Archaeology Image (Zhejiang University), Ministry of Education, Hangzhou Zhejiang 310058, China

Received:2024-08-02 Accepted:2024-10-06 Published:2024-12-31 Online:2024-12-24
First author：CHEN Xiaojiao (1988-), professor, Ph.D. Her main research interests cover human-computer interaction interface, user experience design practice, digital humanities, etc. E-mail：chenxiaojiao@zju.edu.cn
Supported by:
National Natural Science Foundation of China(52205290);Aeronautical Science Foundation(2022Z005076001)

摘要/Abstract

摘要：

用户界面(UI)设计的质量直接影响产品的可用性和用户体验。设计师在UI设计过程中常面临一致性和可访问性问题，这些问题不仅增加了用户的认知负荷，还影响了使用效率。尽管设计师对此有所认识，但目前缺乏全面的知识和工具来进行自动识别和解决这些问题。为此提出了一套全面的UI设计评估准则，涵盖色彩、文本、布局、控件和图标5个关键方面，专门针对UI设计的一致性问题和可访问性问题。基于这套评估准则，提出了针对UI一致性和可访问性评估的提示词模版，以提升大语言模型(LLMs)如GPT-4在UI评估任务中的准确率。此外，开发了基于GPT-4模型的UI评估系统。该UI评估系统能够深入理解UI设计内容，依据评估准则自动检测UI设计问题，并提供针对性改进建议，帮助设计师优化UI设计。实验结果表明，使用提示词模版显著提高了GPT-4模型在UI评估中的准确性。用户研究表明，设计师在设计实践中使用该UI评估系统，可以显著提升UI设计的质量，从而提升产品可用性和用户体验。该系统为设计师提供了一种自动化UI评估工具，为提升UI设计质量提供了新思路。

关键词: 图形用户界面, 大语言模型, UI评估, 一致性, 可访问性

Abstract:

The quality of user interface (UI) design directly impacts product usability and user experience. Designers often face challenges related to consistency and accessibility during the UI design process, increasing cognitive load for users reducing efficiency. Despite awareness of these issues, they currently lack comprehensive knowledge and tools or automatic identification and resolution. To address this challenge, a comprehensive set of UI design evaluation criteria was proposed, covering five key aspects: color, text, layout, control, and icon, specifically targeting consistency and accessibility issues in UI design. Based on these evaluation criteria, a prompt template for evaluating UI consistency and accessibility was developed to enhance the accuracy of large language models (LLMs) like GPT-4 in UI evaluation tasks. Furthermore, a UI evaluation system based on the GPT-4 model was developed. This [26] deeply understood UI design content, automatically detected UI design issues according to the evaluation criteria, and provided targeted improvement suggestions to help designers optimize their UI designs. Experimental results demonstrated that using the prompt template significantly improved the accuracy of GPT-4 in UI evaluations. User studies indicated that employing this UI evaluation system in design practice can significantly enhance the quality of UI designs, thereby boosting product usability and user experience. This system provided designers with an automated UI evaluation tool, offering a new approach to enhancing UI design quality.

Key words: graphical user interface, large language model, UI evaluation, consistency, accessibility

中图分类号:

TP391
TP18

陈晓皎, 束云峰, 汪睿涵, 周佳欢, 陈为. 大语言模型驱动的UI评估系统[J]. 图学学报, 2024, 45(6): 1178-1187.

CHEN Xiaojiao, SHU Yunfeng, WANG Ruihan, ZHOU Jiahuan, CHEN Wei. Large language model powered UI evaluation system[J]. Journal of Graphics, 2024, 45(6): 1178-1187.

图/表 10

图1 系统框架图

Fig. 1 The framework of system

图2 UI截图与UI JSON (部分)

Fig. 2 UI screenshot and UI JSON (partial)

图3 数据流与提示词示例

Fig. 3 Dataflow and prompt example

图4 UI评估工作流程图((a) UI设计原型视图；(b) UI设计评估反馈结果视图)

Fig. 4 UI evaluation workflow diagram ((a) UI design prototype view; (b) UI design evaluation feedback view)

图5 2个UI案例分析

Fig. 5 Two UI case studies analysis

表1 不同输入条件下GPT-4 UI评估结果比较

Table 1 Comparison of GPT-4 UI evaluation results under different input conditions

输入条件	所有平均设计问题数量	有效平均设计问题数量
仅JSON数据输入	6.3	2.9
仅UI图像数据输入	6.5	1.9
JSON数据+UI图像结合输入	6.6	3.8

表2 不同输入条件下GPT-4 UI评估性能指标比较

Table 2 Comparison of performance indicators for GPT-4 UI evaluation under different input conditions

输入条件	Precision	Recall	F1
仅JSON数据输入	0.472	0.525	0.492
仅UI图像输入	0.316	0.354	0.328
JSON数据+UI图像结合输入	0.587	0.683	0.623

表3 不同提示条件下GPT-4 UI评估结果比较

Table 3 Comparison of GPT-4 UI evaluation results under different prompt conditions

提示条件	所有设计问题数量	有效设计问题数量
无提示词	65	37
有提示词	82	62

表4 GPT-4和人类评估师的UI设计评估性能指标比较

Table 4 Comparison of UI Design Evaluation Performance Indicators between GPT-4 and Human Evaluators

评估者	Precision	Recall	F1
GPT-4	0.614	0.362	0.455
人类评估师(平均值)	0.802	0.322	0.459

图6 用户研究实验结果

Fig. 6 User study experiment results

参考文献 26

[1]	GRUDIN J. The case against user interface consistency[J]. Communications of the ACM, 1989, 32(10): 1164-1173.
[2]	ZAINA L A M, FORTES R P M, CASADEI V, et al. Preventing accessibility barriers: guidelines for using user interface design patterns in mobile applications[J]. Journal of Systems and Software, 2022, 186: 111213.
[3]	NIELSEN J. Finding usability problems through heuristic evaluation[C]// The SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 1992: 373-380.
[4]	KARAT C M, CAMPBELL R, FIEGEL T. Comparison of empirical testing and walkthrough methods in user interface evaluation[C]// The SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 1992: 397-404.
[5]	段艳花, 刘子建, 宁铎. 基于眼动技术的TMA界面评估及优化[J]. 图学学报, 2022, 43(4): 745-752.
	DUAN Y H, LIU Z J, NING D. Evaluation and optimization of TMA interface supported eye movement technology[J]. Journal of Graphics, 2022, 43(4): 745-752. (in Chinese)
[6]	郑瑞凌, 张俊松. 脑电时空多特征融合的数字图形界面认知负荷评价方法[J]. 计算机辅助设计与图形学学报, 2020, 32(7): 1062-1069.
	ZHENG R L, ZHANG J S. Assessing cognitive load combining features of time, frequency and spatial domain under digital graphical interface[J]. Journal of Computer-Aided Design & Computer Graphics, 2020, 32(7): 1062-1069. (in Chinese)
[7]	PARK S, KIM W, KIM Y H, et al. Computational approaches for app-to-app retrieval and design consistency check[EB/OL]. [2024-03-02]. https://arxiv.org/abs/2309.10328.
[8]	BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]// The 34th International Conference on Neural Information Processing Systems. Red Hook: Curran Associates Inc., 2020: 159.
[9]	WANG B, LI G, LI Y. Enabling conversational interaction with mobile UI using large language models[C]// 2023 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2023: 432.
[10]	DUAN P T, WARNER J, LI Y, et al. Generating automatic feedback on UI mockups with large language models[C]// 2024 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2024: 6.
[11]	ACHIAM J, ADLER S, AGARWAL S, et al. Gpt-4 technical report[EB/OL]. [2024-03-02]. https://arxiv.org/abs/2303.08774.
[12]	JEFFRIES R, MILLER J R, WHARTON C, et al. User interface evaluation in the real world: a comparison of four techniques[C]// The SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 1991: 119-124.
[13]	ALABOOD L, AMINOLROAYA Z, YIM D, et al. A systematic literature review of the design critique method[J]. Information and Software Technology, 2023, 153: 107081.
[14]	厉向东, 夏涵飞, 单逸飞, 等. 图形用户界面自动生成技术的现状与挑战[J]. 图学学报, 2024, 45(3): 409-421. DOI
	LI X D, XIA H F, SHAN Y F, et al. Opportunities and challenges: automatic generation technologies for graphical user interfaces[J]. Journal of Graphics, 2024, 45(3): 409-421. (in Chinese) DOI
[15]	YANG B, XING Z C, XIA X, et al. Don’t do that! hunting down visual design smells in complex UIs against design guidelines[C]// 2021 IEEE/ACM 43rd International Conference on Software Engineering. New York: IEEE Press, 2021: 761-772.
[16]	LIU Z, CHEN C Y, WANG J J, et al. Nighthawk: fully automated localizing UI display issues via visual understanding[J]. IEEE Transactions on Software Engineering, 2023, 49(1): 403-418.
[17]	YANG B, LI S P. UIGuider: detecting implicit design guidelines using a domain knowledge graph approach[J]. Electronics, 2024, 13(7): 1210.
[18]	GAO S Q, WANG Y H, LIU H X. UiAnalyzer: evaluating whether the UI of apps is at the risk of violating the design conventions in terms of function layout[J]. Expert Systems with Applications, 2024, 239: 122408.
[19]	LU Y W, KNEAREM T, DUTTA S, et al. AI is not enough: a hybrid technical approach to AI adoption in UI linting with heuristics[C]// Extended Abstracts of the CHI Conference on Human Factors in Computing Systems. New York: ACM, 2024: 501.
[20]	XIANG W, ZHU H F, LOU S Q, et al. SimUser: generating usability feedback by simulating various users interacting with mobile applications[C]// 2024 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2024: 9.
[21]	MAHAJAN R, SHNEIDERMAN B. Visual and textual consistency checking tools for graphical user interfaces[J]. IEEE Transactions on Software Engineering, 1997, 23(11): 722-735.
[22]	CHEN X J, TANG X T, ZHAO Y, et al. Evaluating visual consistency of icon usage in across-devices[J]. International Journal of Human-Computer Interaction, 2024, 40(9): 2415-2431.
[23]	刘哲, 王俊杰, 陈春阳, 等. 移动应用程序中用户界面显示缺陷检测的经验研究[EB/OL]. (2024-01-17) [2024-03-29]. http://www.jos.org.cn/1000-9825/7043.htm.
	LIU Z, WANG J J, CHEN C Y, et al. Empirical study on UI display issue detection in mobile applications[EB/OL]. (2024-01-17) [2024-03-29]. http://www.jos.org.cn/1000-9825/7043.htm. (in Chinese)
[24]	KIRKPATRICK A, CONNOR J O, CAMPBELL A, et al. Web content accessibility guidelines (WCAG) 2.1[EB/OL]. (2023-09-21) [2024-02-22]. https://www.w3.org/TR/WCAG21/.
[25]	YAO S Y, ZHAO J, YU D, et al. ReAct: synergizing reasoning and acting in language models[EB/OL]. [2024-02-02]. https://openreview.net/forum?id=WE_vluYUL-X.
[26]	COLLINS C, ANDRIENKO N, SCHRECK T, et al. Guidance in the human-machine analytics process[J]. Visual Informatics, 2018, 2(3): 166-180.

大语言模型驱动的UI评估系统

Large language model powered UI evaluation system

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 26

相关文章 8

编辑推荐

Metrics

本文评价

[1]	徐沛, 黄凯奇. 大模型引导的高效强化学习方法[J]. 图学学报, 2024, 45(6): 1165-1177.
[2]	于晗, 陈治源, 熊熙瑞, 戴原星, 蔡鸿明. 基于检索增强大语言模型的MBSE智能设计方法[J]. 图学学报, 2024, 45(6): 1188-1199.
[3]	许璟琳, 彭阳, 欧金武, 谈骏杰, 舒江鹏, 余芳强. 融合大模型和数字孪生的公共建筑智慧运维系统[J]. 图学学报, 2024, 45(6): 1200-1206.
[4]	厉向东, 夏涵飞, 单逸飞, 阴凯琳, 耿卫东. 图形用户界面自动生成技术的现状与挑战[J]. 图学学报, 2024, 45(3): 409-421.
[5]	蒋灿, 郑哲, 梁雄, 林佳瑞, 马智亮, 陆新征. 大语言模型驱动的交互式建筑设计新范式——基于Rhino7的概念验证[J]. 图学学报, 2024, 45(3): 594-600.
[6]	廖仕敏, 刘仰川, 朱叶晨, 王艳玲, 高欣 . 一种基于 CycleGAN 改进的低剂量 CT 图像增强网络[J]. 图学学报, 2022, 43(4): 570-578.
[7]	罗琪彬 1,2，蔡强 1,2 . 采用双框架生成对抗网络的图像运动模糊盲去除[J]. 图学学报, 2019, 40(6): 1056-1063.
[8]	张岩，何晓波，管涛. 基于多视图几何的位置与遮挡一致性恢复方法研究[J]. 图学学报, 2011, 32(1): 111-118.