The impact of LLM-generated models on novice domain modelers: a comparative experiment

Abstract

In software development, domain models are conceptual blueprints that capture the structure, relationships, and key entities of a problem domain. Automated techniques can support analysts and developers by extracting such models from existing artifacts. However, this is a non-trivial task, especially when the input consists of informal artifacts such as user stories. This paper investigates how providing an initial, automatically generated domain model influences novice developers’ ability to construct their own domain models. We conducted an experiment involving 127 undergraduate students, divided into three groups: one receiving an LLM-generated model that maximizes precision (validity), one receiving an LLM-generated model that boosts recall (completeness), and a control group that did not receive any initial domain model. Our findings show that novices who received an initial LLM-generated model produced more complete class identification in simple user story projects and improved relationship identification accuracy in both simple and complex projects. While initial domain models appear to aid novices in refining domain models effectively, our results also suggest a strong tendency among participants to rely heavily on these initial models. On average, students included 85% of the correct classes and 67% of the incorrect classes from the initial domain models in their own derived models. Such reliance can scaffold novice learning and refinement, but this reliance may limit creativity and hinder the deeper reasoning required to develop robust domain modeling skills.