A novel code generator for graphical user interfaces

被引:1
|
作者
Cai B. [1 ,2 ]
Luo J. [1 ,2 ]
Feng Z. [1 ,2 ]
机构
[1] School of Cyber Science and Engineering, Wuhan University, Wuhan
[2] Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, Wuhan
基金
中国国家自然科学基金;
关键词
D O I
10.1038/s41598-023-46500-6
中图分类号
学科分类号
摘要
Graphical user interfaces (GUIs) are widely used in human–computer interaction, providing a convenient interface for operation. Automating the conversion of GUI design images into source code can significantly reduce the coding workload for front-end developers. Detecting elements in GUI images is a key challenge in achieving automatic GUI code generation and is crucial for tasks such as GUI automation and testing. However, current state-of-the-art methods do not fully consider the unique characteristics of GUI images and elements, and they lack the required high localization accuracy, resulting in low detection accuracy for GUI element boxes. In this paper, we propose GUICG, an automatic GUI code generator that combines deep neural networks with image processing techniques to efficiently detect GUI elements from GUI images and generate front-end code. We empirically investigate various deep learning approaches and image processing methods for GUI component detection. Based on a comprehensive understanding of their performance and characteristics, we design GUICG by fusing image processing with a deep learning-based target detection model, achieving state-of-the-art performance. GUICG outperforms existing methods in accuracy and F1 score for component detection tasks, while producing human-readable code with a logical structure. Furthermore, we conduct an ablation study to quantitatively assess the impact of each key element in GUICG. © 2023, The Author(s).
引用
收藏
相关论文
共 50 条
  • [1] Graphical user interfaces
    Martinez, Wendy L.
    WILEY INTERDISCIPLINARY REVIEWS-COMPUTATIONAL STATISTICS, 2011, 3 (02): : 119 - 133
  • [2] GRAPHICAL USER INTERFACES
    MAHONEY, K
    BYTE, 1989, 14 (10): : 36 - 36
  • [3] GRAPHICAL INTERFACE CODE GENERATOR
    DIAS, GP
    GOMES, PJ
    RAFAEL, JA
    PINTO, AJ
    EUROPEAN JOURNAL OF NUCLEAR MEDICINE, 1992, 19 (08): : 640 - 640
  • [4] Challenges and opportunities for improving code-based testing of graphical user interfaces
    Karam, Marcel R.
    Dascalu, Sergiu M.
    Hazime, Rami H.
    JOURNAL OF COMPUTATIONAL METHODS IN SCIENCES AND ENGINEERING, 2006, 6 (5-6) : S379 - S388
  • [5] Graphical user interfaces for bioconductor
    Ophir, R
    BRIEFINGS IN BIOINFORMATICS, 2004, 5 (03) : 297 - 299
  • [6] Graphical User Interfaces for R
    Valero-Mora, Pedro M.
    Ledesma, Ruben D.
    JOURNAL OF STATISTICAL SOFTWARE, 2012, 49 (01): : 1 - 8
  • [7] Differencing Graphical User Interfaces
    Grechanik, Mark
    Mao, Chi Wu
    Baisal, Ankush
    Hossain, B. M. Mainul
    Rosenblum, David S.
    2018 IEEE INTERNATIONAL CONFERENCE ON SOFTWARE QUALITY, RELIABILITY AND SECURITY (QRS 2018), 2018, : 203 - 214
  • [8] Generic graphical user interfaces
    Achten, P
    van Eekelen, M
    Plasmeijer, R
    IMPLEMENTATION OF FUNCTIONAL LANGUAGES, 2004, 3145 : 152 - 167
  • [9] How to test graphical user interfaces
    Dabóczi, T
    Kollár, I
    Simon, G
    Megyeri, T
    IEEE INSTRUMENTATION & MEASUREMENT MAGAZINE, 2003, 6 (03) : 27 - 33
  • [10] A Semantic Model for Graphical User Interfaces
    Krishnaswami, Neelakantan R.
    Benton, Nick
    ICFP 11 - PROCEEDINGS OF THE 2011 ACM SIGPLAN: INTERNATIONAL CONFERENCE ON FUNCTIONAL PROGRAMMING, 2011, : 45 - +