Opportunities and challenges: automatic generation technologies for graphical user interfaces

doi:10.11996/JG.j.2095-302X.2024030409

Abstract

Abstract:

As emerging technologies like big data and artificial intelligence continue to evolve and find applications, a trend towards algorithm-driven automation and intelligent graphical user interface (GUI) design is rapidly transforming traditional manual methods. The need for automated GUI generation arises from the diversity of human-computer interaction devices, increasing functional complexity, and the demand for personalization. This approach not only addresses these challenges but also establishes a new paradigm in intelligent, collaborative human-computer interface design. To further expand developers’ and researchers’ understanding and application of cutting-edge technology methods and application cases in automatic GUI generation, this paper delved into the intersection of human-computer interaction and artificial intelligence, shedding light on the latest advancements in automated GUI generation and intelligent evaluation methods. The current technological approaches, algorithmic models, design methods, and evaluation frameworks for automatic GUI generation were analyzed. Advancements in GUI auto-generation technology based on artificial intelligence generated content (AIGC) were specifically highlighted, along with the new challenges posed to existing user interface design paradigms. Furthermore, the development trends and opportunities in the field of automatic GUI generation were summarized.

Key words: human-computer interaction, intelligent user interface design, automatic graphical user interface generation, interface design standards, interface evaluation methods

CLC Number:

TP391

LI Xiangdong, XIA Hanfei, SHAN Yifei, YIN Kailin, GENG Weidong. Opportunities and challenges: automatic generation technologies for graphical user interfaces[J]. Journal of Graphics, 2024, 45(3): 409-421.

Figures/Tables 14

Fig. 1 Overview of the paper structure

Fig. 2 Flowchart of graphic user interface layout generation with neural machine translator

Table 1 Datasets of graphic user interface

名称	类型	数量	平台
Rico^[32]	软件界面图	72 k	安卓端界面
Enrico^[33]	软件界面图	1 460	安卓端界面
Screen2words^[30]	软件界面及短语	22 417	安卓端界面
Sketch2Code^[41]	手绘草图和高保真界面图	3 398	网页端界面
Pix2code^[18,42]	图形界面及合成界面	苹果端：2 600 000 安卓端：21 000 000 网页端：310 000	苹果端安卓端网页端界面
UISketch^[34]	手绘草图	17 979	跨平台界面元素
Syn^[35]	低保真手绘草图	125 000	跨平台界面元素
SynZ^[36]	合成的手绘界面草图	175 377	跨平台界面元素

Fig. 3 Single-page graphic designs with various specifications

Fig. 4 Comparison of works ((a) Designer-based; (b) Algorithm-based)

Fig. 5 Comparison of design works ((a) Original material; (b) Designer-based; (c) Luban-based; (d) Vinci-based)

Fig. 6 Poster layouts designed by intelligent design systems ((a) The first poster; (b) The second poster)

Fig. 7 Software user interfaces generated by neural design network ((a) The first UI; (b) The second UI)

Fig. 8 Timeline of text-to-image technology development (Green: generative adversarial networks; Yellow: auto-regressive methods; Red: diffusion models)

Fig. 9 Comparison of three methods for user interface generation ((a) LayoutVAE; (b) LayoutTransformer; (c) VNT)

Fig. 10 Comparison of three methods for user interface completion ((a) GNN; (b) Transformer; (c) kNN)

Fig. 11 GPT-4 based graphical user interface skeleton generation

Fig. 12 E-commence user interfaces generated by AIGC (Midjourney)

Fig. 13 E-commence user interfaces generated by AIGC (DALL-E)

References 128

[1]	STONE D, JARRETT C, WOODROFFE M, et al. User interface design and evaluation[M]. San Francisco: Morgan Kaufmann, 2005: 3-24.
[2]	JANSEN B J. The graphical user interface[J]. ACM SIGCHI Bulletin, 1998, 30(2): 22-26.
[3]	MARTINEZ W L. Graphical user interfaces[J]. WIREs Computational Statistics, 2011, 3(2): 119-133.
[4]	GUO S N, JIN Z C, SUN F L, et al. Vinci: an intelligent graphic design system for generating advertising posters[C]// The 2021 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2021: 1-17.
[5]	ZHAO T M, CHEN C Y, LIU Y N, et al. GUIGAN: learning to generate GUI designs using generative adversarial networks[C]// 2021 IEEE/ACM 43rd International Conference on Software Engineering. New York: IEEE Press, 2021: 748-760.
[6]	GE X F. Android GUI search using hand-drawn sketches[C]// 2019 IEEE/ACM 41st International Conference on Software Engineering:Companion Proceedings. New York: IEEE Press, 2019: 141-143.
[7]	CHEN J S, CHEN C Y, XING Z C, et al. Wireframe-based UI design search through image autoencoder[EB/OL]. [2023-01-12]. https://doi.org/10.1145/3391613.
[8]	QASIM I, AZAM F, ANWAR M W, et al. Mobile user interface development techniques: a systematic literature review[C]// 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference. New York: IEEE Press, 2019: 1029-1034.
[9]	SHI Y, SHANG M Y, QI Z Q. Intelligent layout generation based on deep generative models: a comprehensive survey[J]. Information Fusion, 2023, 100: 101940.
[10]	MYERS B, HUDSON S E, PAUSCH R. Past, present, and future of user interface software tools[J]. ACM Transactions on Computer-Human Interaction, 2000, 7(1): 3-28.
[11]	HIRSCHBERG J, MANNING C D. Advances in natural language processing[J]. Science, 2015, 349(6245): 261-266. DOI PMID
[12]	赵轶男. 车载终端界面的触觉反馈技术[D]. 长春: 吉林大学, 2023.
	ZHAO Y N. Haptic feedback technology for automotive terminal interface[D]. Changchun: Jilin University, 2023 (in Chinese).
[13]	王韫, 何丽雯, 王党校. 智能家居中的触觉交互体验[J]. 包装工程, 2022, 43(16): 37-49, 108.
	WANG Y, HE L W, WANG D X. Haptic interaction experience in smart homes[J]. Packaging Engineering, 2022, 43(16): 37-49, 108 (in Chinese).
[14]	HABLER F, PEISKER M, HENZE N. Differences between smart speakers and graphical user interfaces for music search considering gender effects[C]// The 18th International Conference on Mobile and Ubiquitous Multimedia. New York: ACM, 2019: 1-7.
[15]	KOLTHOFF K. Automatic generation of graphical user interface prototypes from unrestricted natural language requirements[C]// 2019 34th IEEE/ACM International Conference on Automated Software Engineering. New York: IEEE Press, 2020: 1234-1237.
[16]	ZHAO N X, KIM N W, HERMAN L M, et al. ICONATE: automatic compound icon generation and ideation[C]// 2020 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2020: 1-13.
[17]	HUSSAIN J, UL HASSAN A, MUHAMMAD BILAL H S, et al. Model-based adaptive user interface based on context and user experience evaluation[J]. Journal on Multimodal User Interfaces, 2018, 12(1): 1-16.
[18]	KARRAY F, ALEMZADEH M, ABOU SALEH J, et al. Human-computer interaction: overview on state of the art[J]. International Journal on Smart Sensing and Intelligent Systems, 2008, 1(1): 137-159.
[19]	SAKINAH S I, FADHLILLAH H S, AZURAT A, et al. Proposed user interface generation for software product lines engineering[C]// 2018 International Conference on Advanced Computer Science and Information Systems. New York: IEEE Press, 2019: 481-486.
[20]	GAULKE W, ZIEGLER J. Using profiled ontologies to leverage model driven user interface generation[EB/OL]. [2023-02-01]. https://doi.org/10.1145/2774225.2775070.
[21]	RUIZ J, SERRAL E, SNOECK M. UI-GEAR: user interface generation prEview capable to adapt in real-time[EB/OL]. [2023-02-01]. https://doi.org/10.5220/0006115402770284.
[22]	SIROHI P, AGARWAL A, MAHESHWARI P. A survey on augmented virtual reality: applications and future directions[C]// 2020 7th International Conference on Information Technology Trends. New York: IEEE Press, 2021: 99-106.
[23]	赵沁平. 虚拟现实综述[J]. 中国科学: F辑: 信息科学, 2009, 39(1): 2-46.
	ZHAO Q P. Overview of virtual reality[J]. Science in China: Series F: Information Sciences, 2009, 39(1): 2-46 (in Chinese).
[24]	MACKINLAY J, CARD S K, ROBERTSON G G. A semantic analysis of the design space of input devices[J]. Human-Computer Interaction, 1990, 5(2-3): 145-190.
[25]	朱淼良, 姚远, 蒋云良. 增强现实综述[J]. 中国图象图形学报, 2004, 9(7): 767-774.
	ZHU M L, YAO Y, JIANG Y L. A survey on augmented reality[J]. Journal of Image and Graphics, 2004, 9(7): 767-774. (in Chinese).
[26]	JEFFRI N F S, RAMBLI D R A. Guidelines for the interface design of AR systems for manual assembly[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3385378.3385389.
[27]	XIE Y, HUANG D, WANG J, et al. CanvasEmb: learning layout representation with large-scale pre-training for graphic design[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3474085.3475541.
[28]	FENG S D, MA S Y, YU J Z, et al. Auto icon: an automated code generation tool for icon desiigns in UI development[C]// IUI ’21: The 26th Intemational Conference on Intelligent User Interface. New York: ACM, 2021:59-69.
[29]	CHEN C Y, SU T, MENG G Z, et al. From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation[C]// 2018 IEEE/ACM 40th International Conference on Software Engineering. New York: IEEE Press, 2018: 665-676.
[30]	WANG B, LI G, ZHOU X, et al. Screen2Words: automatic mobile UI summarization with multimodal learning[C]// UIST ’21: The 34th Annual ACM Symposium on User Interface Software and Technology. New York: ACM, 2021: 498-510.
[31]	ZHANG T, LIU Y, GAO J, et al. Deep learning-based mobile application isomorphic GUI identification for automated robotic testing[J]. IEEE Software, 2020, 37(4): 67-74.
[32]	DEKA B, HUANG Z F, FRANZEN C, et al. Rico: a mobile app dataset for building data-driven design applications[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3126594.3126651.
[33]	LEIVA L A, HOTA A, OULASVIRTA A. Enrico: a dataset for topic modeling of mobile UI designs[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3406324.3410710.
[34]	PANDIAN V P S, SULERI S, JARKE P D M. UISketch: a large-scale dataset of UI element sketches[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3411764.3445784.
[35]	PANDIAN V P S, SULERI S, JARKE M. Syn: synthetic dataset for training UI element detector from lo-fi sketches[C]// The 25th International Conference on Intelligent User Interfaces Companion. New York: ACM, 2020: 79-80.
[36]	PANDIAN V P S, SULERI S, JARKE M. SynZ: enhanced synthetic dataset for training UI element detectors[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3397482.3450725.
[37]	LI Y, LI G, HE L H, et al. Widget captioning: generating natural language description for mobile user interface elements[EB/OL]// The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [ 2023-02-01]. https://doi.org/10.48550/arXiv.2010.04295.
[38]	LIU Z, CHEN C Y, WANG J J, et al. Owl eyes: spotting UI display issues via visual understanding[C]// 2020 35th IEEE/ACM International Conference on Automated Software Engineering. New York: IEEE Press, 2020: 398-409.
[39]	CHANDEL R S, SHARMA S, KAUR S, et al. Smart watches: a review of evolution in bio-medical sector[J]. Materials Today: Proceedings, 2022, 50: 1053-1066.
[40]	BUNIAN S, LI K, JEMMALI C, et al. VINS: visual search for mobile user interface design[C]// 2021 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2021: 1-14.
[41]	JAIN V, AGRAWAL P, BANGA S, et al. Sketch2Code: transformation of sketches to UI in real-time using deep neural network[EB/OL]. [2023-01-23]. https://arxiv.org/abs/1910.08930.pdf.
[42]	BELTRAMELLI T. Pix2code: generating code from a graphical user interface screenshot[C]// The ACM SIGCHI Symposium on Engineering Interactive Computing Systems. New York: ACM, 2018: 1-6.
[43]	RUIZ J, SERRAL E, SNOECK M. Unifying functional user interface design principles[J]. International Journal of Human-Computer Interaction, 2021, 37(1): 47-67.
[44]	JOHNSON J. Designing with the mind in mind: simple guide to understanding user interface design guidelines[M]. 3rd ed. Burlington: Morgan Kaufmann, 2020: 11-14.
[45]	NACHEVA R. Principles of user interface design: important rules that every designer should follow[J]. Izvestia Journal of the Union of Scientists - Varna. Economic Sciences Series, 2015, 1: 140-149.
[46]	SHNEIDERMAN B, PLAISANT C, COHEN M, et al. Designing the user interface: strategies for effective human-computer interaction[M]. 6th ed. London: Pearson, 2016: 57-78.
[47]	NIELSEN J. Ten usability heuristics[EB/OL]. [2023-02-01]. https://www.nngroup.com/articles/ten-usability-heuristics/.
[48]	RAO NAIDU V, SRINIVAS S, AL RAISI M, et al. Evaluation of hypermedia tools in terms of usability heuristics for English language teaching[J]. Arab World English Journal, 2021(2): 133-149.
[49]	JIANG R, GU Z Y. Current theoretical developments and applications of fitts’ law: a literature review[C]// International Conference on Applied Human Factors and Ergonomics. Cham: Springer, 2020: 753-760.
[50]	KAPROS E. Fitts’ law as an education resource for human-computer interaction in computer science curricula[J]. International Online Journal of Primary Education, 2018, 7(1): 28-39.
[51]	WANG H F, WANG J L, TANG Q H. A review of application of affordance theory in information systems[J]. Journal of Service Science and Management, 2018, 11(1): 56-70.
[52]	SCHOOP E, ZHOU X, LI G, et al. Predicting and explaining mobile UI tappability with vision modeling and saliency analysis[C]// 2022 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2022: 1-21.
[53]	LIANG Y X. Application of Gestalt psychology in product human-machine Interface design[J]. IOP Conference Series: Materials Science and Engineering, 2018, 392: 062054.
[54]	LI B Y, CAO Y. Research on layout design of main interface of stadium monitoring system based on gestalt psychology[M]// Human Interface and the Management of Information. Visual Information and Knowledge Management. Cham: Springer International Publishing, 2019: 44-55.
[55]	蔡松霏. 移动互联网应用界面设计中的视觉统一性研究[D]. 杭州: 中国美术学院, 2018.
	CAI S F. Visual unity in user interface of mobile Internet applications[D]. Hangzhou: China Academy of Art, 2018 (in Chinese).
[56]	SHELLEY C. The nature of simplicity in Apple design[J]. The Design Journal, 2015, 18(3): 439-456.
[57]	EL-SHERBINY H F. Realism and 3D graphics in UI designs and social media platforms (Trend of 2020/21)[J]. Journal of Arts & Architecture Research Studies, 2022, 3(5): 138-53.
[58]	MACIK M. Automatic user interface generation[D]. Prague: Czech Technical University, 2016.
[59]	ERTL D. Semi-automatic multimodal user interface generation[C]// The 1st ACM SIGCHI symposium on Engineering interactive computing systems. New York: ACM, 2009: 321-324.
[60]	O’DONOVAN P, AGARWALA A, HERTZMANN A. Learning layouts for single-PageGraphic designs[J]. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(8): 1200-1213.
[61]	O'DONOVAN P, AGARWALA A, HERTZMANN A. DesignScape: design with interactive layout suggestions[C]// The 33rd Annual ACM Conference on Human Factors in Computing Systems. New York: ACM, 2015: 1221-1224.
[62]	RANEBURGER D, POPP R, VANDERDONCKT J. An automated layout approach for model-driven WIMP-UI generation[C]// The 4th ACM SIGCHI Symposium on Engineering Interactive Computing systems. New York: ACM, 2012: 91-100.
[63]	YANG X Y, MEI T, XU Y Q, et al. Automatic generation of visual-textual presentation layout[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2016, 12(2): 33: 1-33: 22.
[64]	TABATA S, YOSHIHARA H, MAEDA H, et al. Automatic layout generation for graphical design magazines[C]// SIGGRAPH ’19: ACM SIGGRAPH 2019 Posters. New York: ACM, 2019: 1-2.
[65]	BRÜCKNER L. Graphical User Interface auto-completion with element constraints[D]. Malaga: ETSI_Informatica, 2020.
[66]	VEMPATI S, MALAYIL K T, SRUTHI V, et al. Enabling hyper-personalisation: automated ad creative generation and ranking for fashion e-commerce[C]// Fashion Recommender Systems. Cham: Springer, 2020: 25-48.
[67]	REBELO S M, MARTINS T, BICKER J, et al. Exploring automatic fitness evaluation for evolutionary typesetting[C]// The 13th Conference on Creativity and Cognition. New York: ACM, 2021: 1-9.
[68]	LI J N, YANG J M, HERTZMANN A, et al. LayoutGAN: generating graphic layouts with wireframe discriminators[EB/OL]. [2023-01-13]. https://arxiv.org/abs/1901.06767.pdf.
[69]	KIKUCHI K, SIMO-SERRA E, OTANI M, et al. Constrained graphic layout generation via latent optimization[C]// The 29th ACM International Conference on Multimedia. New York: ACM, 2021: 88-96.
[70]	LEE H Y, JIANG L, ESSA I, et al. Neural design network: graphic layout generation with constraints[C]// European Conference on Computer Vision. Cham: Springer, 2020: 491-506.
[71]	ZHENG X R, QIAO X T, CAO Y, et al. Content-aware generative modeling of graphic design layouts[EB/OL]. [2023-02-02]. https://doi.org/10.1145/3306346.3322971.
[72]	ZHOU M, XU C C, MA Y, et al. Composition-aware graphic layout GAN for visual-textual presentation designs[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2205.00303.pdf.
[73]	ZHANG C, ZHANG C, ZHANG M, et al. Text-to-image diffusion model in generative ai: A survey[EB/OL]. [2023-05-01]. https://doi.org/10.48550/arXiv.2303.07909.
[74]	QIAO T T, ZHANG J, XU D Q, et al. MirrorGAN: learning text-to-image generation by redescription[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1505-1514.
[75]	LI J N, YANG J M, ZHANG J M, et al. Attribute-conditioned layout GAN for automatic graphic design[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(10): 4039-4048.
[76]	GAJJAR N, PANDIAN V P S, SULERI S, et al. Akin: generating UI wireframes from UI design patterns using deep learning[EB/OL]. [2023-01-11]. https://doi.org/10.1145/3397482.3450727.
[77]	KAINDL H. High-level interaction design with discourse models for automated web GUI generation[C]// International Conference on Web Engineering. Cham: Springer, 2021: 542-546.
[78]	ENGEL J, MÄRTIN C, FORBRIG P. Practical aspects of pattern-supported model-driven user interface generation[C]// International Conference on Human-Computer Interaction. Cham: Springer, 2017: 397-414.
[79]	CHEN S, FAN L L, SU T, et al. Automated cross-platform GUI code generation for mobile apps[C]// 2019 IEEE 1st International Workshop on Artificial Intelligence for Mobile. New York: IEEE Press, 2019: 13-16.
[80]	MISTRY M, APTE A, GHODAKE V, et al. Machine learning based user interface generation[C]// International Conference on Intelligent Computing, Information and Control Systems. Cham: Springer, 2020: 453-460.
[81]	ZHANG W, LUAN S M, TIAN L Q. A rapid combined model for automatic generating web UI codes[J]. Wireless Communications and Mobile Computing, 2022, 2022: 1-10.
[82]	PANG X W, ZHOU Y Q, LI P C, et al. A novel syntax-aware automatic graphics code generation with attention-based deep neural network[J]. Journal of Network and Computer Applications, 2020, 161: 102636.
[83]	MORAN K, BERNAL-CÁRDENAS C, CURCIO M, et al. Machine learning-based prototyping of graphical user interfaces for mobile apps[J]. IEEE Transactions on Software Engineering, 2020, 46(2): 196-221.
[84]	HEITKÖTTER H, MAJCHRZAK T A, KUCHEN H. Cross-platform model-driven development of mobile applications with md²[C]// The 28th Annual ACM Symposium on Applied Computing. New York: ACM, 2013: 526-533.
[85]	CAO Y H, LI S Y, LIU YX, et al. A comprehensive survey of AI-generated content (AIGC): a history of generative AI from GAN to ChatGPT[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2303.04226.pdf.
[86]	WU J Y, GAN W S, CHEN Z F, et al. AI-generated content (AIGC): a survey[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2304.06632.pdf.
[87]	CHEN C, FU J, LYU L J. A pathway towards responsible AI generated content[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2303.01325.pdf.
[88]	VAN DIS E A M, BOLLEN J, ZUIDEMA W, et al. ChatGPT: five priorities for research[J]. Nature, 2023, 614(7947): 224-226.
[89]	XIA M, ZHOU Y C. Research on UI design and optimization of digital media based on artificial intelligence[J]. Journal of Sensors, 2022, 2022: 1-17.
[90]	GUPTA K, LAZAROW J, ACHILLE A, et al. LayoutTransformer: layout generation and completion with self-attention[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2022: 984-994.
[91]	JYOTHI A A, DURAND T, HE J W, et al. LayoutVAE: stochastic scene layout generation from a label set[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2020: 9894-9903.
[92]	ARROYO D M, POSTELS J, TOMBARI F. Variational transformer networks for layout generation[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 13637-13647.
[93]	KIRMANI A R. Artificial intelligence-enabled science poetry[J]. ACS Energy Letters, 2023, 8(1): 574-576.
[94]	LUND B, TING W. Chatting about ChatGPT: how may AI and GPT impact academia and libraries?[EB/OL]. [2023-06-01]. https://www.emerald.com/insight/content/doi/10.1108/LHTN-01-2023-0009/full/html.
[95]	ADESSO G. Towards the ultimate brain: exploring scientific discovery with ChatGPT AI[J]. AI Magazine, 2023, 44(3): 328-342.
[96]	BORJI A. Generated faces in the wild: quantitative comparison of stable diffusion, midjourney and DALL-E 2[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2210.00586.pdf.
[97]	ROMBACH R, BLATTMANN A, LORENZ D, et al. High-resolution image synthesis with latent diffusion models[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 10684-10695.
[98]	WEI J L, COURBIS A L, LAMBOLAIS T, et al. Boosting GUI prototyping with diffusion models[C]// 2023 IEEE 31st International Requirements Engineering Conference. New York: IEEE Press, 2023: 275-280.
[99]	XU J P, ZHANG X L, LI H, et al. Is everyone an artist? A study on user experience of AI-based painting system[J]. Applied Sciences, 2023, 13(11): 6496.
[100]	ZHANG C N, ZHANG C S, ZHENG S, et al. A complete survey on generative AI (AIGC): is ChatGPT from GPT-4 to GPT-5 all You need? [EB/OL]. [2023-01-13]. https://arxiv.org/abs/2303.11717.pdf. 2023.
[101]	ZHANG L M, RAO A Y, AGRAWALA M. Adding conditional control to text-to-image diffusion models[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2302.05543.pdf.
[102]	JEFFRIES R, MILLER J R, WHARTON C, et al. User interface evaluation in the real world: a comparison of four techniques[C]/ The SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 1991: 119-124.
[103]	郑瑞凌, 张俊松. 脑电时空多特征融合的数字图形界面认知负荷评价方法[J]. 计算机辅助设计与图形学学报, 2020, 32(7): 1062-1069.
	ZHENG R L, ZHANG J S. Assessing cognitive load combining features of time, frequency and spatial domain under digital graphical interface[J]. Journal of Computer- Aided Design & Computer Graphics, 2020, 32(7): 1062-1069 (in Chinese).
[104]	王洪宝, 王金, 支锦亦, 等. 基于RAMSIS仿真的高速列车驾驶界面人机工效评估[J]. 机械设计, 2020, 37(1): 128-134. DOI
	WANG H B, WANG J, ZHI J Y, et al. Ergonomic evaluation of high-speed train driving interface based on RAMSIS simulation[J]. Journal of Machine Design, 2020, 37(1): 128-134 (in Chinese).
[105]	姜钧译. 手机游戏用户体验测量方法研究[D]. 沈阳: 东北大学, 2019.
	JIANG J Y. Measurement method for user experience of mobile games[D]. Shenyang: Northeastern University, 2019 (in Chinese).
[106]	NIELSEN J, MOLICH R. Heuristic evaluation of user interfaces[EB/OL]. [2022-11-01]. https://doi.org/10.1145/97243.97281.
[107]	ALOMARI H W, RAMASAMY V, KIPER J D, et al. A User Interface (UI) and User eXperience (UX) evaluation framework for cyberlearning environments in computer science and software engineering education[J]. Heliyon, 2020, 6(5): e03917.
[108]	HERMAWATI S, LAWSON G. Establishing usability heuristics for heuristics evaluation in a specific domain: is there a consensus?[J]. Applied Ergonomics, 2016, 56: 34-51. DOI PMID
[109]	ISLAM M N, BOUWMAN H, ISLAM A K M N. Evaluating web and mobile user interfaces with semiotics: an empirical study[J]. IEEE Access, 2020, 8: 84396-84414.
[110]	WICHANSKY A M. Usability testing in 2000 and beyond[J]. Ergonomics, 2000, 43(7): 998-1006. PMID
[111]	NURWULAN N R, ZAHIRAH K F. User interface evaluation of a real-time gimmick tracking[J]. Journal of Physics: Conference Series, 2021, 1882(1): 012123.
[112]	ZHANG W J, FELTNER D, SHIRLEY J, et al. Enhancement and application of a UAV control interface evaluation technique: modified GEDIS-UAV[J]. ACM Transactions on Human-Robot Interaction, 2020, 9(2): 14:1-14:20,
[113]	HUILCAPI-COLLANTES C, MARTÍN A H, HERNÁNDEZ-RAMOS J P. Pedagogical and user interface usability evaluation of an educational mobile app that promotes visual literacy[C]// TEEM’20:Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality. New York: ACM, 2020: 315-321.
[114]	FANG Y M, LIN C. The usability testing of VR interface for tourism apps[J]. Applied Sciences, 2019, 9(16): 3215.
[115]	ROHRER C P, WENDT J, SAURO J, et al. Practical usability rating by experts (PURE): a pragmatic approach for scoring product usability[C]// 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. New York: ACM, 2016: 786-795.
[116]	段艳花, 刘子建, 宁铎. 基于眼动技术的TMA界面评估及优化[J]. 图学学报, 2022, 43(4): 745-752.
	DUAN Y H, LIU Z J, NING D. Evaluation and optimization of TMA interface supported eye movement technology[J]. Journal of Graphics, 2022, 43(4): 745-752 (in Chinese).
[117]	GOLDBERG J H, KOTVAL X P. Computer interface evaluation using eye movements: methods and constructs[J]. International Journal of Industrial Ergonomics, 1999, 24(6): 631-645.
[118]	王诗傲. 基于眼动实验的工业设计服务网站界面评估体系研究[D]. 西安: 西安工程大学, 2017.
	WANG S A. The research about industrial design website evaluation system based on eye-tracking experiment[D]. Xi'an: Xi'an Polytechnic University, 2017 (in Chinese).
[119]	王刚, 王子凡. 基于眼动追踪的智能电视界面可用性评估研究[J]. 包装工程, 2022, 43(18): 65-71.
	WANG G, WANG Z F. Usability evaluation of smart TV interface based on eye tracking[J]. Packaging Engineering, 2022, 43(18): 65-71 (in Chinese).
[120]	CHENG S W, DEY A K. I see, you design: user interface intelligent design system with eye tracking and interactive genetic algorithm[J]. CCF Transactions on Pervasive Computing and Interaction, 2019, 1(3): 224-236.
[121]	ZHANG J Y, CHANG D N, ZHANG Z. Review on the application of eye-tracking technology in usability evaluation of E-government apps[C]// 2021 IEEE International Conference on Industrial Engineering and Engineering Management. New York: IEEE Press, 2022: 1646-1650.
[122]	蔡船, 邓丽, 陈波, 等. 脑电技术在工业设计中的应用[J]. 机械设计与制造工程, 2017, 46(10): 80-84.
	CAI C, DENG L, CHEN B, et al. Application of EEG technology in industrial design[J]. Machine Design and Manufacturing Engineering, 2017, 46(10): 80-84 (in Chinese).
[123]	牛亚峰. 基于脑电技术的数字界面可用性评价方法研究[D]. 南京: 东南大学, 2016.
	NIU Y F. Usability evaluation methods research of digital interface based on brain electrical technology[D]. Nanjing: Southeast University, 2016 (in Chinese).
[124]	SULIKOWSKI P, ZDZIEBKO T. Deep learning-enhanced framework for performance evaluation of a recommending interface with varied recommendation position and intensity based on eye-tracking equipment data processing[J]. Electronics, 2020, 9(2): 266.
[125]	WANG W G, SHEN J B. Deep visual attention prediction[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2018, 27(5): 2368-2378.
[126]	YAN S Y, WEI Y Y, TRAN C C. Evaluation and prediction mental workload in user interface of maritime operations using eye response[J]. International Journal of Industrial Ergonomics, 2019, 71: 117-127.
[127]	窦金花. 面向包容性的人机交互界面评估与优化方法研究[D]. 北京: 北京科技大学, 2021.
	DOU J H. Research on evaluation and optimization methods for inclusiveness-oriented human-computer interface[D]. Beijing: University of Science and Technology Beijing, 2021 (in Chinese).
[128]	GONZALEZ-CALLEROS J, OSTERLOH J P, FEIL R, et al. Automated UI evaluation based on a cognitive architecture and UsiXML[J]. Science of Computer Programming, 2014, 86: 43-57.