Real Time Detection of Mobile Graphical User Interface Elements Using Convolutional Neural Networks

In this work, we model the Graphical User Interface (GUI) detection challenge as an object detection problem from Computer Vision (CV) domain. Based on our literature review, we identified some works with similar proposals but suffering from reproducibility and comparability problems. Thus, we propose to mitigate these problems by creating a standardized dataset that can be used for training and evaluating CV algorithms in Mobile GUI. For this purpose, we use Rico's Android application screen collection and semantic annotation of GUI elements and labelled them using the standard Microsoft COCO format for object detection. Finally, we split the dataset into three main challenges: 1) clickable and nonclickable elements; 2) interface components detection; and 3) icons detection. We trained a baseline algorithm considered state-of-theart on real-time object detection from the YOLO family. Finally, we present quantitative results for the three proposed challenges.