图形用户界面自动生成技术的现状与挑战

doi:10.11996/JG.j.2095-302X.2024030409

图学学报 ›› 2024, Vol. 45 ›› Issue (3): 409-421.DOI: 10.11996/JG.j.2095-302X.2024030409

图形用户界面自动生成技术的现状与挑战

厉向东(), 夏涵飞, 单逸飞, 阴凯琳, 耿卫东

浙江大学计算机科学与技术学院，浙江杭州 310000

收稿日期:2023-08-31 接受日期:2023-12-04 出版日期:2024-06-30 发布日期:2024-06-06
第一作者:厉向东(1984-)，男，副教授，博士。主要研究方向为智能人机界面和跨设备交互。E-mail：axli@zju.edu.cn
基金资助:
国家自然科学基金重点项目(61972346);国家自然科学基金项目(92148205);浙江省重点研发计划项目(2021C03128);浙江省重点研发计划项目(2022C03103);浙江省“尖兵”计划项目(2023C01045)

Opportunities and challenges: automatic generation technologies for graphical user interfaces

LI Xiangdong(), XIA Hanfei, SHAN Yifei, YIN Kailin, GENG Weidong

College of Computer Science and Technology, Zhejiang University, Hangzhou Zhejiang 310000, China

Received:2023-08-31 Accepted:2023-12-04 Published:2024-06-30 Online:2024-06-06
First author：LI Xiangdong (1984-), associate professor, Ph.D. His main research interests cover intelligent human-machine interface and cross-device interaction. E-mail：axli@zju.edu.cn
Supported by:
Key Project of National Natural Science Foundation of China(61972346);National Natural Science Foundation of China(92148205);Zhejiang Provincial Key R&D Program(2021C03128);Zhejiang Provincial Key R&D Program(2022C03103);“Pioneer” R&D Program of Zhejiang Province(2023C01045)

摘要/Abstract

摘要：

随着大数据、人工智能等新兴技术的不断发展和应用，算法模型驱动的图形用户界面自动化和智能化趋势正以前所未有的速度改变着传统的依赖人工绘制的图形用户界面设计方式。用户界面自动生成不仅是解决人机交互设备多样化、功能复杂化、需求个性化等挑战的必然选择，也为人机协同的智能用户界面设计提供了新范式。为了进一步拓展开发人员和研究人员对图形用户界面自动生成前沿技术方法和应用案例的理解及应用，阐述了当前人机交互和人工智能交叉领域下的图形用户界面自动生成及智能评估相关技术的现状，分析了目前图形用户界面自动生成的技术路径、算法模型、设计方法以及评估框架，特别介绍了基于人工智能内容生成技术(AIGC)的图形用户界面自动生成技术进展及其对现有用户界面设计范式带来的新挑战，并总结了图形用户界面自动生成技术的发展趋势和机遇。

关键词: 人机交互, 智能用户界面设计, 图形用户界面自动生成, 界面设计规范, 界面评估方法

Abstract:

As emerging technologies like big data and artificial intelligence continue to evolve and find applications, a trend towards algorithm-driven automation and intelligent graphical user interface (GUI) design is rapidly transforming traditional manual methods. The need for automated GUI generation arises from the diversity of human-computer interaction devices, increasing functional complexity, and the demand for personalization. This approach not only addresses these challenges but also establishes a new paradigm in intelligent, collaborative human-computer interface design. To further expand developers’ and researchers’ understanding and application of cutting-edge technology methods and application cases in automatic GUI generation, this paper delved into the intersection of human-computer interaction and artificial intelligence, shedding light on the latest advancements in automated GUI generation and intelligent evaluation methods. The current technological approaches, algorithmic models, design methods, and evaluation frameworks for automatic GUI generation were analyzed. Advancements in GUI auto-generation technology based on artificial intelligence generated content (AIGC) were specifically highlighted, along with the new challenges posed to existing user interface design paradigms. Furthermore, the development trends and opportunities in the field of automatic GUI generation were summarized.

Key words: human-computer interaction, intelligent user interface design, automatic graphical user interface generation, interface design standards, interface evaluation methods

中图分类号:

TP391

厉向东, 夏涵飞, 单逸飞, 阴凯琳, 耿卫东. 图形用户界面自动生成技术的现状与挑战[J]. 图学学报, 2024, 45(3): 409-421.

LI Xiangdong, XIA Hanfei, SHAN Yifei, YIN Kailin, GENG Weidong. Opportunities and challenges: automatic generation technologies for graphical user interfaces[J]. Journal of Graphics, 2024, 45(3): 409-421.

图/表 14

图1 总体内容示意图

Fig. 1 Overview of the paper structure

图2 神经机器翻译器的图形用户界面布局生成流程示意图

Fig. 2 Flowchart of graphic user interface layout generation with neural machine translator

表1 图形用户界面主流数据集

Table 1 Datasets of graphic user interface

名称	类型	数量	平台
Rico^[32]	软件界面图	72 k	安卓端界面
Enrico^[33]	软件界面图	1 460	安卓端界面
Screen2words^[30]	软件界面及短语	22 417	安卓端界面
Sketch2Code^[41]	手绘草图和高保真界面图	3 398	网页端界面
Pix2code^[18,42]	图形界面及合成界面	苹果端：2 600 000 安卓端：21 000 000 网页端：310 000	苹果端安卓端网页端界面
UISketch^[34]	手绘草图	17 979	跨平台界面元素
Syn^[35]	低保真手绘草图	125 000	跨平台界面元素
SynZ^[36]	合成的手绘界面草图	175 377	跨平台界面元素

图3 不同规格的单页图形设计

Fig. 3 Single-page graphic designs with various specifications

图4 比较作品((a)设计师绘制；(b)算法生成)

Fig. 4 Comparison of works ((a) Designer-based; (b) Algorithm-based)

图5 比较设计结果((a)原素材；(b)设计师作品；(c)鹿班作品；(d) Vinci作品)

Fig. 5 Comparison of design works ((a) Original material; (b) Designer-based; (c) Luban-based; (d) Vinci-based)

图6 自动生成的海报排版结果((a)海报1；(b)海报2)

Fig. 6 Poster layouts designed by intelligent design systems ((a) The first poster; (b) The second poster)

图7 基于神经设计网络的软件界面设计((a) 界面1；(b)界面2)

Fig. 7 Software user interfaces generated by neural design network ((a) The first UI; (b) The second UI)

图8 文生图技术的发展历程(绿色：基于生成式对抗网络；黄色：自回归方法；红色：基于扩散模型)

Fig. 8 Timeline of text-to-image technology development (Green: generative adversarial networks; Yellow: auto-regressive methods; Red: diffusion models)

图9 3种界面生成方法的结果对比

Fig. 9 Comparison of three methods for user interface generation ((a) LayoutVAE; (b) LayoutTransformer; (c) VNT)

图10 3种界面补全方法的结果对比

Fig. 10 Comparison of three methods for user interface completion ((a) GNN; (b) Transformer; (c) kNN)

图11 基于GPT-4的图形用户界面骨架生成结果

Fig. 11 GPT-4 based graphical user interface skeleton generation

图12 AIGC (Midjourney) 生成的线上购物程序用户界面

Fig. 12 E-commence user interfaces generated by AIGC (Midjourney)

图13 AIGC (DALL-E) 生成的线上购物程序用户界面

Fig. 13 E-commence user interfaces generated by AIGC (DALL-E)

参考文献 128

[1]	STONE D, JARRETT C, WOODROFFE M, et al. User interface design and evaluation[M]. San Francisco: Morgan Kaufmann, 2005: 3-24.
[2]	JANSEN B J. The graphical user interface[J]. ACM SIGCHI Bulletin, 1998, 30(2): 22-26.
[3]	MARTINEZ W L. Graphical user interfaces[J]. WIREs Computational Statistics, 2011, 3(2): 119-133.
[4]	GUO S N, JIN Z C, SUN F L, et al. Vinci: an intelligent graphic design system for generating advertising posters[C]// The 2021 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2021: 1-17.
[5]	ZHAO T M, CHEN C Y, LIU Y N, et al. GUIGAN: learning to generate GUI designs using generative adversarial networks[C]// 2021 IEEE/ACM 43rd International Conference on Software Engineering. New York: IEEE Press, 2021: 748-760.
[6]	GE X F. Android GUI search using hand-drawn sketches[C]// 2019 IEEE/ACM 41st International Conference on Software Engineering:Companion Proceedings. New York: IEEE Press, 2019: 141-143.
[7]	CHEN J S, CHEN C Y, XING Z C, et al. Wireframe-based UI design search through image autoencoder[EB/OL]. [2023-01-12]. https://doi.org/10.1145/3391613.
[8]	QASIM I, AZAM F, ANWAR M W, et al. Mobile user interface development techniques: a systematic literature review[C]// 2018 IEEE 9th Annual Information Technology, Electronics and Mobile Communication Conference. New York: IEEE Press, 2019: 1029-1034.
[9]	SHI Y, SHANG M Y, QI Z Q. Intelligent layout generation based on deep generative models: a comprehensive survey[J]. Information Fusion, 2023, 100: 101940.
[10]	MYERS B, HUDSON S E, PAUSCH R. Past, present, and future of user interface software tools[J]. ACM Transactions on Computer-Human Interaction, 2000, 7(1): 3-28.
[11]	HIRSCHBERG J, MANNING C D. Advances in natural language processing[J]. Science, 2015, 349(6245): 261-266. DOI PMID
[12]	赵轶男. 车载终端界面的触觉反馈技术[D]. 长春: 吉林大学, 2023.
	ZHAO Y N. Haptic feedback technology for automotive terminal interface[D]. Changchun: Jilin University, 2023 (in Chinese).
[13]	王韫, 何丽雯, 王党校. 智能家居中的触觉交互体验[J]. 包装工程, 2022, 43(16): 37-49, 108.
	WANG Y, HE L W, WANG D X. Haptic interaction experience in smart homes[J]. Packaging Engineering, 2022, 43(16): 37-49, 108 (in Chinese).
[14]	HABLER F, PEISKER M, HENZE N. Differences between smart speakers and graphical user interfaces for music search considering gender effects[C]// The 18th International Conference on Mobile and Ubiquitous Multimedia. New York: ACM, 2019: 1-7.
[15]	KOLTHOFF K. Automatic generation of graphical user interface prototypes from unrestricted natural language requirements[C]// 2019 34th IEEE/ACM International Conference on Automated Software Engineering. New York: IEEE Press, 2020: 1234-1237.
[16]	ZHAO N X, KIM N W, HERMAN L M, et al. ICONATE: automatic compound icon generation and ideation[C]// 2020 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2020: 1-13.
[17]	HUSSAIN J, UL HASSAN A, MUHAMMAD BILAL H S, et al. Model-based adaptive user interface based on context and user experience evaluation[J]. Journal on Multimodal User Interfaces, 2018, 12(1): 1-16.
[18]	KARRAY F, ALEMZADEH M, ABOU SALEH J, et al. Human-computer interaction: overview on state of the art[J]. International Journal on Smart Sensing and Intelligent Systems, 2008, 1(1): 137-159.
[19]	SAKINAH S I, FADHLILLAH H S, AZURAT A, et al. Proposed user interface generation for software product lines engineering[C]// 2018 International Conference on Advanced Computer Science and Information Systems. New York: IEEE Press, 2019: 481-486.
[20]	GAULKE W, ZIEGLER J. Using profiled ontologies to leverage model driven user interface generation[EB/OL]. [2023-02-01]. https://doi.org/10.1145/2774225.2775070.
[21]	RUIZ J, SERRAL E, SNOECK M. UI-GEAR: user interface generation prEview capable to adapt in real-time[EB/OL]. [2023-02-01]. https://doi.org/10.5220/0006115402770284.
[22]	SIROHI P, AGARWAL A, MAHESHWARI P. A survey on augmented virtual reality: applications and future directions[C]// 2020 7th International Conference on Information Technology Trends. New York: IEEE Press, 2021: 99-106.
[23]	赵沁平. 虚拟现实综述[J]. 中国科学: F辑: 信息科学, 2009, 39(1): 2-46.
	ZHAO Q P. Overview of virtual reality[J]. Science in China: Series F: Information Sciences, 2009, 39(1): 2-46 (in Chinese).
[24]	MACKINLAY J, CARD S K, ROBERTSON G G. A semantic analysis of the design space of input devices[J]. Human-Computer Interaction, 1990, 5(2-3): 145-190.
[25]	朱淼良, 姚远, 蒋云良. 增强现实综述[J]. 中国图象图形学报, 2004, 9(7): 767-774.
	ZHU M L, YAO Y, JIANG Y L. A survey on augmented reality[J]. Journal of Image and Graphics, 2004, 9(7): 767-774. (in Chinese).
[26]	JEFFRI N F S, RAMBLI D R A. Guidelines for the interface design of AR systems for manual assembly[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3385378.3385389.
[27]	XIE Y, HUANG D, WANG J, et al. CanvasEmb: learning layout representation with large-scale pre-training for graphic design[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3474085.3475541.
[28]	FENG S D, MA S Y, YU J Z, et al. Auto icon: an automated code generation tool for icon desiigns in UI development[C]// IUI ’21: The 26th Intemational Conference on Intelligent User Interface. New York: ACM, 2021:59-69.
[29]	CHEN C Y, SU T, MENG G Z, et al. From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation[C]// 2018 IEEE/ACM 40th International Conference on Software Engineering. New York: IEEE Press, 2018: 665-676.
[30]	WANG B, LI G, ZHOU X, et al. Screen2Words: automatic mobile UI summarization with multimodal learning[C]// UIST ’21: The 34th Annual ACM Symposium on User Interface Software and Technology. New York: ACM, 2021: 498-510.
[31]	ZHANG T, LIU Y, GAO J, et al. Deep learning-based mobile application isomorphic GUI identification for automated robotic testing[J]. IEEE Software, 2020, 37(4): 67-74.
[32]	DEKA B, HUANG Z F, FRANZEN C, et al. Rico: a mobile app dataset for building data-driven design applications[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3126594.3126651.
[33]	LEIVA L A, HOTA A, OULASVIRTA A. Enrico: a dataset for topic modeling of mobile UI designs[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3406324.3410710.
[34]	PANDIAN V P S, SULERI S, JARKE P D M. UISketch: a large-scale dataset of UI element sketches[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3411764.3445784.
[35]	PANDIAN V P S, SULERI S, JARKE M. Syn: synthetic dataset for training UI element detector from lo-fi sketches[C]// The 25th International Conference on Intelligent User Interfaces Companion. New York: ACM, 2020: 79-80.
[36]	PANDIAN V P S, SULERI S, JARKE M. SynZ: enhanced synthetic dataset for training UI element detectors[EB/OL]. [2023-02-01]. https://doi.org/10.1145/3397482.3450725.
[37]	LI Y, LI G, HE L H, et al. Widget captioning: generating natural language description for mobile user interface elements[EB/OL]// The 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [ 2023-02-01]. https://doi.org/10.48550/arXiv.2010.04295.
[38]	LIU Z, CHEN C Y, WANG J J, et al. Owl eyes: spotting UI display issues via visual understanding[C]// 2020 35th IEEE/ACM International Conference on Automated Software Engineering. New York: IEEE Press, 2020: 398-409.
[39]	CHANDEL R S, SHARMA S, KAUR S, et al. Smart watches: a review of evolution in bio-medical sector[J]. Materials Today: Proceedings, 2022, 50: 1053-1066.
[40]	BUNIAN S, LI K, JEMMALI C, et al. VINS: visual search for mobile user interface design[C]// 2021 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2021: 1-14.
[41]	JAIN V, AGRAWAL P, BANGA S, et al. Sketch2Code: transformation of sketches to UI in real-time using deep neural network[EB/OL]. [2023-01-23]. https://arxiv.org/abs/1910.08930.pdf.
[42]	BELTRAMELLI T. Pix2code: generating code from a graphical user interface screenshot[C]// The ACM SIGCHI Symposium on Engineering Interactive Computing Systems. New York: ACM, 2018: 1-6.
[43]	RUIZ J, SERRAL E, SNOECK M. Unifying functional user interface design principles[J]. International Journal of Human-Computer Interaction, 2021, 37(1): 47-67.
[44]	JOHNSON J. Designing with the mind in mind: simple guide to understanding user interface design guidelines[M]. 3rd ed. Burlington: Morgan Kaufmann, 2020: 11-14.
[45]	NACHEVA R. Principles of user interface design: important rules that every designer should follow[J]. Izvestia Journal of the Union of Scientists - Varna. Economic Sciences Series, 2015, 1: 140-149.
[46]	SHNEIDERMAN B, PLAISANT C, COHEN M, et al. Designing the user interface: strategies for effective human-computer interaction[M]. 6th ed. London: Pearson, 2016: 57-78.
[47]	NIELSEN J. Ten usability heuristics[EB/OL]. [2023-02-01]. https://www.nngroup.com/articles/ten-usability-heuristics/.
[48]	RAO NAIDU V, SRINIVAS S, AL RAISI M, et al. Evaluation of hypermedia tools in terms of usability heuristics for English language teaching[J]. Arab World English Journal, 2021(2): 133-149.
[49]	JIANG R, GU Z Y. Current theoretical developments and applications of fitts’ law: a literature review[C]// International Conference on Applied Human Factors and Ergonomics. Cham: Springer, 2020: 753-760.
[50]	KAPROS E. Fitts’ law as an education resource for human-computer interaction in computer science curricula[J]. International Online Journal of Primary Education, 2018, 7(1): 28-39.
[51]	WANG H F, WANG J L, TANG Q H. A review of application of affordance theory in information systems[J]. Journal of Service Science and Management, 2018, 11(1): 56-70.
[52]	SCHOOP E, ZHOU X, LI G, et al. Predicting and explaining mobile UI tappability with vision modeling and saliency analysis[C]// 2022 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2022: 1-21.
[53]	LIANG Y X. Application of Gestalt psychology in product human-machine Interface design[J]. IOP Conference Series: Materials Science and Engineering, 2018, 392: 062054.
[54]	LI B Y, CAO Y. Research on layout design of main interface of stadium monitoring system based on gestalt psychology[M]// Human Interface and the Management of Information. Visual Information and Knowledge Management. Cham: Springer International Publishing, 2019: 44-55.
[55]	蔡松霏. 移动互联网应用界面设计中的视觉统一性研究[D]. 杭州: 中国美术学院, 2018.
	CAI S F. Visual unity in user interface of mobile Internet applications[D]. Hangzhou: China Academy of Art, 2018 (in Chinese).
[56]	SHELLEY C. The nature of simplicity in Apple design[J]. The Design Journal, 2015, 18(3): 439-456.
[57]	EL-SHERBINY H F. Realism and 3D graphics in UI designs and social media platforms (Trend of 2020/21)[J]. Journal of Arts & Architecture Research Studies, 2022, 3(5): 138-53.
[58]	MACIK M. Automatic user interface generation[D]. Prague: Czech Technical University, 2016.
[59]	ERTL D. Semi-automatic multimodal user interface generation[C]// The 1st ACM SIGCHI symposium on Engineering interactive computing systems. New York: ACM, 2009: 321-324.
[60]	O’DONOVAN P, AGARWALA A, HERTZMANN A. Learning layouts for single-PageGraphic designs[J]. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(8): 1200-1213.
[61]	O'DONOVAN P, AGARWALA A, HERTZMANN A. DesignScape: design with interactive layout suggestions[C]// The 33rd Annual ACM Conference on Human Factors in Computing Systems. New York: ACM, 2015: 1221-1224.
[62]	RANEBURGER D, POPP R, VANDERDONCKT J. An automated layout approach for model-driven WIMP-UI generation[C]// The 4th ACM SIGCHI Symposium on Engineering Interactive Computing systems. New York: ACM, 2012: 91-100.
[63]	YANG X Y, MEI T, XU Y Q, et al. Automatic generation of visual-textual presentation layout[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2016, 12(2): 33: 1-33: 22.
[64]	TABATA S, YOSHIHARA H, MAEDA H, et al. Automatic layout generation for graphical design magazines[C]// SIGGRAPH ’19: ACM SIGGRAPH 2019 Posters. New York: ACM, 2019: 1-2.
[65]	BRÜCKNER L. Graphical User Interface auto-completion with element constraints[D]. Malaga: ETSI_Informatica, 2020.
[66]	VEMPATI S, MALAYIL K T, SRUTHI V, et al. Enabling hyper-personalisation: automated ad creative generation and ranking for fashion e-commerce[C]// Fashion Recommender Systems. Cham: Springer, 2020: 25-48.
[67]	REBELO S M, MARTINS T, BICKER J, et al. Exploring automatic fitness evaluation for evolutionary typesetting[C]// The 13th Conference on Creativity and Cognition. New York: ACM, 2021: 1-9.
[68]	LI J N, YANG J M, HERTZMANN A, et al. LayoutGAN: generating graphic layouts with wireframe discriminators[EB/OL]. [2023-01-13]. https://arxiv.org/abs/1901.06767.pdf.
[69]	KIKUCHI K, SIMO-SERRA E, OTANI M, et al. Constrained graphic layout generation via latent optimization[C]// The 29th ACM International Conference on Multimedia. New York: ACM, 2021: 88-96.
[70]	LEE H Y, JIANG L, ESSA I, et al. Neural design network: graphic layout generation with constraints[C]// European Conference on Computer Vision. Cham: Springer, 2020: 491-506.
[71]	ZHENG X R, QIAO X T, CAO Y, et al. Content-aware generative modeling of graphic design layouts[EB/OL]. [2023-02-02]. https://doi.org/10.1145/3306346.3322971.
[72]	ZHOU M, XU C C, MA Y, et al. Composition-aware graphic layout GAN for visual-textual presentation designs[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2205.00303.pdf.
[73]	ZHANG C, ZHANG C, ZHANG M, et al. Text-to-image diffusion model in generative ai: A survey[EB/OL]. [2023-05-01]. https://doi.org/10.48550/arXiv.2303.07909.
[74]	QIAO T T, ZHANG J, XU D Q, et al. MirrorGAN: learning text-to-image generation by redescription[C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 1505-1514.
[75]	LI J N, YANG J M, ZHANG J M, et al. Attribute-conditioned layout GAN for automatic graphic design[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(10): 4039-4048.
[76]	GAJJAR N, PANDIAN V P S, SULERI S, et al. Akin: generating UI wireframes from UI design patterns using deep learning[EB/OL]. [2023-01-11]. https://doi.org/10.1145/3397482.3450727.
[77]	KAINDL H. High-level interaction design with discourse models for automated web GUI generation[C]// International Conference on Web Engineering. Cham: Springer, 2021: 542-546.
[78]	ENGEL J, MÄRTIN C, FORBRIG P. Practical aspects of pattern-supported model-driven user interface generation[C]// International Conference on Human-Computer Interaction. Cham: Springer, 2017: 397-414.
[79]	CHEN S, FAN L L, SU T, et al. Automated cross-platform GUI code generation for mobile apps[C]// 2019 IEEE 1st International Workshop on Artificial Intelligence for Mobile. New York: IEEE Press, 2019: 13-16.
[80]	MISTRY M, APTE A, GHODAKE V, et al. Machine learning based user interface generation[C]// International Conference on Intelligent Computing, Information and Control Systems. Cham: Springer, 2020: 453-460.
[81]	ZHANG W, LUAN S M, TIAN L Q. A rapid combined model for automatic generating web UI codes[J]. Wireless Communications and Mobile Computing, 2022, 2022: 1-10.
[82]	PANG X W, ZHOU Y Q, LI P C, et al. A novel syntax-aware automatic graphics code generation with attention-based deep neural network[J]. Journal of Network and Computer Applications, 2020, 161: 102636.
[83]	MORAN K, BERNAL-CÁRDENAS C, CURCIO M, et al. Machine learning-based prototyping of graphical user interfaces for mobile apps[J]. IEEE Transactions on Software Engineering, 2020, 46(2): 196-221.
[84]	HEITKÖTTER H, MAJCHRZAK T A, KUCHEN H. Cross-platform model-driven development of mobile applications with md²[C]// The 28th Annual ACM Symposium on Applied Computing. New York: ACM, 2013: 526-533.
[85]	CAO Y H, LI S Y, LIU YX, et al. A comprehensive survey of AI-generated content (AIGC): a history of generative AI from GAN to ChatGPT[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2303.04226.pdf.
[86]	WU J Y, GAN W S, CHEN Z F, et al. AI-generated content (AIGC): a survey[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2304.06632.pdf.
[87]	CHEN C, FU J, LYU L J. A pathway towards responsible AI generated content[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2303.01325.pdf.
[88]	VAN DIS E A M, BOLLEN J, ZUIDEMA W, et al. ChatGPT: five priorities for research[J]. Nature, 2023, 614(7947): 224-226.
[89]	XIA M, ZHOU Y C. Research on UI design and optimization of digital media based on artificial intelligence[J]. Journal of Sensors, 2022, 2022: 1-17.
[90]	GUPTA K, LAZAROW J, ACHILLE A, et al. LayoutTransformer: layout generation and completion with self-attention[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2022: 984-994.
[91]	JYOTHI A A, DURAND T, HE J W, et al. LayoutVAE: stochastic scene layout generation from a label set[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2020: 9894-9903.
[92]	ARROYO D M, POSTELS J, TOMBARI F. Variational transformer networks for layout generation[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 13637-13647.
[93]	KIRMANI A R. Artificial intelligence-enabled science poetry[J]. ACS Energy Letters, 2023, 8(1): 574-576.
[94]	LUND B, TING W. Chatting about ChatGPT: how may AI and GPT impact academia and libraries?[EB/OL]. [2023-06-01]. https://www.emerald.com/insight/content/doi/10.1108/LHTN-01-2023-0009/full/html.
[95]	ADESSO G. Towards the ultimate brain: exploring scientific discovery with ChatGPT AI[J]. AI Magazine, 2023, 44(3): 328-342.
[96]	BORJI A. Generated faces in the wild: quantitative comparison of stable diffusion, midjourney and DALL-E 2[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2210.00586.pdf.
[97]	ROMBACH R, BLATTMANN A, LORENZ D, et al. High-resolution image synthesis with latent diffusion models[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 10684-10695.
[98]	WEI J L, COURBIS A L, LAMBOLAIS T, et al. Boosting GUI prototyping with diffusion models[C]// 2023 IEEE 31st International Requirements Engineering Conference. New York: IEEE Press, 2023: 275-280.
[99]	XU J P, ZHANG X L, LI H, et al. Is everyone an artist? A study on user experience of AI-based painting system[J]. Applied Sciences, 2023, 13(11): 6496.
[100]	ZHANG C N, ZHANG C S, ZHENG S, et al. A complete survey on generative AI (AIGC): is ChatGPT from GPT-4 to GPT-5 all You need? [EB/OL]. [2023-01-13]. https://arxiv.org/abs/2303.11717.pdf. 2023.
[101]	ZHANG L M, RAO A Y, AGRAWALA M. Adding conditional control to text-to-image diffusion models[EB/OL]. [2023-01-13]. https://arxiv.org/abs/2302.05543.pdf.
[102]	JEFFRIES R, MILLER J R, WHARTON C, et al. User interface evaluation in the real world: a comparison of four techniques[C]/ The SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 1991: 119-124.
[103]	郑瑞凌, 张俊松. 脑电时空多特征融合的数字图形界面认知负荷评价方法[J]. 计算机辅助设计与图形学学报, 2020, 32(7): 1062-1069.
	ZHENG R L, ZHANG J S. Assessing cognitive load combining features of time, frequency and spatial domain under digital graphical interface[J]. Journal of Computer- Aided Design & Computer Graphics, 2020, 32(7): 1062-1069 (in Chinese).
[104]	王洪宝, 王金, 支锦亦, 等. 基于RAMSIS仿真的高速列车驾驶界面人机工效评估[J]. 机械设计, 2020, 37(1): 128-134. DOI
	WANG H B, WANG J, ZHI J Y, et al. Ergonomic evaluation of high-speed train driving interface based on RAMSIS simulation[J]. Journal of Machine Design, 2020, 37(1): 128-134 (in Chinese).
[105]	姜钧译. 手机游戏用户体验测量方法研究[D]. 沈阳: 东北大学, 2019.
	JIANG J Y. Measurement method for user experience of mobile games[D]. Shenyang: Northeastern University, 2019 (in Chinese).
[106]	NIELSEN J, MOLICH R. Heuristic evaluation of user interfaces[EB/OL]. [2022-11-01]. https://doi.org/10.1145/97243.97281.
[107]	ALOMARI H W, RAMASAMY V, KIPER J D, et al. A User Interface (UI) and User eXperience (UX) evaluation framework for cyberlearning environments in computer science and software engineering education[J]. Heliyon, 2020, 6(5): e03917.
[108]	HERMAWATI S, LAWSON G. Establishing usability heuristics for heuristics evaluation in a specific domain: is there a consensus?[J]. Applied Ergonomics, 2016, 56: 34-51. DOI PMID
[109]	ISLAM M N, BOUWMAN H, ISLAM A K M N. Evaluating web and mobile user interfaces with semiotics: an empirical study[J]. IEEE Access, 2020, 8: 84396-84414.
[110]	WICHANSKY A M. Usability testing in 2000 and beyond[J]. Ergonomics, 2000, 43(7): 998-1006. PMID
[111]	NURWULAN N R, ZAHIRAH K F. User interface evaluation of a real-time gimmick tracking[J]. Journal of Physics: Conference Series, 2021, 1882(1): 012123.
[112]	ZHANG W J, FELTNER D, SHIRLEY J, et al. Enhancement and application of a UAV control interface evaluation technique: modified GEDIS-UAV[J]. ACM Transactions on Human-Robot Interaction, 2020, 9(2): 14:1-14:20,
[113]	HUILCAPI-COLLANTES C, MARTÍN A H, HERNÁNDEZ-RAMOS J P. Pedagogical and user interface usability evaluation of an educational mobile app that promotes visual literacy[C]// TEEM’20:Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality. New York: ACM, 2020: 315-321.
[114]	FANG Y M, LIN C. The usability testing of VR interface for tourism apps[J]. Applied Sciences, 2019, 9(16): 3215.
[115]	ROHRER C P, WENDT J, SAURO J, et al. Practical usability rating by experts (PURE): a pragmatic approach for scoring product usability[C]// 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. New York: ACM, 2016: 786-795.
[116]	段艳花, 刘子建, 宁铎. 基于眼动技术的TMA界面评估及优化[J]. 图学学报, 2022, 43(4): 745-752.
	DUAN Y H, LIU Z J, NING D. Evaluation and optimization of TMA interface supported eye movement technology[J]. Journal of Graphics, 2022, 43(4): 745-752 (in Chinese).
[117]	GOLDBERG J H, KOTVAL X P. Computer interface evaluation using eye movements: methods and constructs[J]. International Journal of Industrial Ergonomics, 1999, 24(6): 631-645.
[118]	王诗傲. 基于眼动实验的工业设计服务网站界面评估体系研究[D]. 西安: 西安工程大学, 2017.
	WANG S A. The research about industrial design website evaluation system based on eye-tracking experiment[D]. Xi'an: Xi'an Polytechnic University, 2017 (in Chinese).
[119]	王刚, 王子凡. 基于眼动追踪的智能电视界面可用性评估研究[J]. 包装工程, 2022, 43(18): 65-71.
	WANG G, WANG Z F. Usability evaluation of smart TV interface based on eye tracking[J]. Packaging Engineering, 2022, 43(18): 65-71 (in Chinese).
[120]	CHENG S W, DEY A K. I see, you design: user interface intelligent design system with eye tracking and interactive genetic algorithm[J]. CCF Transactions on Pervasive Computing and Interaction, 2019, 1(3): 224-236.
[121]	ZHANG J Y, CHANG D N, ZHANG Z. Review on the application of eye-tracking technology in usability evaluation of E-government apps[C]// 2021 IEEE International Conference on Industrial Engineering and Engineering Management. New York: IEEE Press, 2022: 1646-1650.
[122]	蔡船, 邓丽, 陈波, 等. 脑电技术在工业设计中的应用[J]. 机械设计与制造工程, 2017, 46(10): 80-84.
	CAI C, DENG L, CHEN B, et al. Application of EEG technology in industrial design[J]. Machine Design and Manufacturing Engineering, 2017, 46(10): 80-84 (in Chinese).
[123]	牛亚峰. 基于脑电技术的数字界面可用性评价方法研究[D]. 南京: 东南大学, 2016.
	NIU Y F. Usability evaluation methods research of digital interface based on brain electrical technology[D]. Nanjing: Southeast University, 2016 (in Chinese).
[124]	SULIKOWSKI P, ZDZIEBKO T. Deep learning-enhanced framework for performance evaluation of a recommending interface with varied recommendation position and intensity based on eye-tracking equipment data processing[J]. Electronics, 2020, 9(2): 266.
[125]	WANG W G, SHEN J B. Deep visual attention prediction[J]. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2018, 27(5): 2368-2378.
[126]	YAN S Y, WEI Y Y, TRAN C C. Evaluation and prediction mental workload in user interface of maritime operations using eye response[J]. International Journal of Industrial Ergonomics, 2019, 71: 117-127.
[127]	窦金花. 面向包容性的人机交互界面评估与优化方法研究[D]. 北京: 北京科技大学, 2021.
	DOU J H. Research on evaluation and optimization methods for inclusiveness-oriented human-computer interface[D]. Beijing: University of Science and Technology Beijing, 2021 (in Chinese).
[128]	GONZALEZ-CALLEROS J, OSTERLOH J P, FEIL R, et al. Automated UI evaluation based on a cognitive architecture and UsiXML[J]. Science of Computer Programming, 2014, 86: 43-57.

图形用户界面自动生成技术的现状与挑战

Opportunities and challenges: automatic generation technologies for graphical user interfaces

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 14

参考文献 128

相关文章 15

编辑推荐

Metrics

本文评价

[1]	高越, 韩红雷. 基于可发光设备的观众席画面展示系统[J]. 图学学报, 2023, 44(4): 784-793.
[2]	段艳花, 刘子建, 宁铎. 基于眼动技术的 TMA 界面评估及优化[J]. 图学学报, 2022, 43(4): 745-752.
[3]	马宁, 王亚辉. 智能汽车座舱人机交互任务复杂度分析方法[J]. 图学学报, 2022, 43(2): 356-360.
[4]	王秋惠, 王雅馨. 医院消杀机器人作业安全与交互设计策略[J]. 图学学报, 2022, 43(1): 172-180.
[5]	黄进, 张浩, 田丰. 面向动态交互场景的计算模型[J]. 图学学报, 2021, 42(3): 359-366.
[6]	柳有权 , 王愿超 , 徐琨 , 刘正雄 , 黄攀峰 . 基于混合现实的远程协同式装配维修引导[J]. 图学学报, 2021, 42(2): 216-221.
[7]	赵建军, 黄竣鹏, 陈俊良. 基于 Leap Motion 的电影前期预演人机交互方法[J]. 图学学报, 2021, 42(1): 71-78.
[8]	郑明钰, 李家和, 张晗, 骆岩林, 申佳丽, 朱小明. 支持力反馈的沉浸式物理学习环境的构建 [J]. 图学学报, 2021, 42(1): 79-86.
[9]	王尔卓1，袁翔1，李士岩2. 智能家居场景中会话智能体主动交互设计研究[J]. 图学学报, 2020, 41(4): 658-666.
[10]	林莹莹，蔡睿凡，朱雨真，唐祥峻，金小刚. 基于Leap Motion 的虚拟现实陶艺体验系统[J]. 图学学报, 2020, 41(1): 57-65.
[11]	戴莎 1,2，司伟鑫 2,3，钱银玲 2，郑睿 2，王琼 2，徐东亮 1，彭延军 4，王平安 3,2 . 虚拟显微白内障手术系统人机交互接口设计[J]. 图学学报, 2019, 40(3): 565-573.
[12]	王苗辉 1，李艳 1，宋武 2，黄梅英 1，陈莉 1 . 基于科学实验的人机交互界面设计研究 ——以格力空调为例[J]. 图学学报, 2019, 40(1): 181-185.
[13]	孙辉，吕健，寸文哲. VR 系统信息可视化模型[J]. 图学学报, 2018, 39(2): 317-326.
[14]	马丽莎，吕健，潘伟杰，单军军，平正强. 基于眼动模式的隐式意图识别分类方法研究[J]. 图学学报, 2017, 38(3): 332-340.
[15]	王宁，余隋怀，周宪，肖琳臻. 人机交互界面中形状特征的视觉显著度计算[J]. 图学学报, 2016, 37(4): 514-518.