Locating requirements in backlog items: Content analysis and experiments with large language models

Abstract

Context: As agile development has become mainstream, requirements are increasingly managed via issue tracking systems (ITSs). These systems provide a single point of access to the product and sprint backlogs, bugs, ideas, and tasks for the development team. ITSs do not clearly separate requirements from work items. Objective: We first tackle a knowledge problem concerning how requirements are formulated in ITSs, including their categorization and granularity, the presence of multiple requirements, and the existence of a motivation. Second, to assist practitioners in finding requirements in poorly organized ITSs without changing their way of working, we investigate the potential of automated techniques for identifying and classifying requirements in backlog items. Method: Through quantitative content analysis, we analyze 1,636 product backlog items sampled from fourteen projects. To explore automated techniques for identifying requirements, we experiment with large language models (LLMs) due to their recent significance in NLP. Results: The labeling of backlog items is largely inconsistent, and user-oriented functional requirements are the prevalent category. A backlog item often contains multiple requirements with different levels of granularity. The experiments with LLMs reveal that encoder-only models (BERT and RoBERTa) are most suitable for extracting and classifying requirements in backlog items compared to decoder-only models (Llama 3, Mistral 7B and ChatGPT with GPT 4). Conclusion: We reveal knowledge and patterns about requirements documentation in ITSs, leading to a better empirical understanding of Agile RE. The experimental results with LLMs provide the foundation for developing automated, unobtrusive tools that identify and classify requirements in ITSs.