Deriving Domain Models from User Stories: Human vs. Machines

Domain models play a crucial role in software development, as they provide means for communication among stakeholders, for eliciting requirements, and for representing the information structure behind a database scheme or at the basis of model-driven development. However, creating such models is a tedious activity and automated support may assist in obtaining an initial domain model that can later be enriched by human analysts. In this paper, we propose an experimental comparison of the effectiveness of various approaches for deriving domain models from a given set of user stories. We contrast human derivation with machine derivation; for the latter, we compare (i) the Visual Narrator: an existing rule-based NLP approach; (ii) a machine-learning classifier that we feature engineered; and (iii) a generative AI approach that we constructed via prompt engineering. Based on a benchmark dataset that consists of nine collections of user stories and corresponding domain models, the evaluation indicates that no approach matches human performance, although a tuned version of the machine learning approach comes close. To better understand the results, we qualitatively analyze them and identify differences in the types of false positives as well as other factors that affect performance.