Multiclass Classification of UML Diagrams from Images Using Deep Learning

Unified Modeling Language (UML) diagrams are a recognized standard modeling language for representing design of software systems. For academic research, large cases containing UML diagrams are needed. One of the challenges in collecting such datasets is automatically determining whether an image is a UML diagram or not and what type of UML diagram an image contains. In this work, we collected UML diagrams from open datasets and manually labeled them into 10 types of UML diagrams (i.e. class diagrams, activity diagrams, use case diagrams, sequence diagrams, communication diagrams, component diagrams, deployment diagrams, object diagrams, package diagrams, and state machine diagrams) and non-UML images. We evaluated the performance of seven popular neural network architectures using transfer learning on the dataset of 4706 images, including 700 class diagrams, 454 activity diagrams, 651 use case diagrams, 706 sequence diagrams, 204 communication diagrams, 208 component diagrams, 287 deployment diagrams, 207 object diagrams, 246 package diagrams, 323 state machine diagrams, and 720 non-UML images, respectively. We also proposed our neural network architecture for multiclass classification of UML diagrams. The experiment results show that Xception achieved the best performance amongst the algorithms we evaluated with a precision of 93.03%, a recall of 92.44%, and an F1-score of 92.73%. Moreover, it is possible to develop small and almost the same efficient neural network architectures, that our proposed architecture has the least parameters (around 2.4 millions) and spends the least time per image (0.0135s per image using graphics processing unit) for classifying UML diagrams with a precision of 91.25%, a recall of 90.34%, and an F1-score of 90.79%.