One Size Does Not Fit All: On the Role of Batch Size in Classifying Requirements with LLMs

Abstract

Automated requirements classification is a widely explored research topic in requirements engineering. In particular, the distinction between functional and non-functional requirements has received considerable attention. Recently, Large Language Models (LLMs) have demonstrated potential in automating requirement classification tasks. Although existing research emphasizes effective prompting strategies, it provides limited evaluation of how many requirements should be processed within a single prompt sequence as a batch to optimize classifier performance. The batch size is relevant when computational resources are constrained, as minimizing the number of LLM calls becomes essential. Moreover, batching requirements may provide the model with additional contextual information, potentially improving the classification performance. Therefore, this study investigates the impact of batch size on classification performance. We assess how three locally deployable models, Llama3-8B, Gemma3-12B, and DeepSeek-Distill-Qwen 14B, perform in classifying requirements according to their functional and quality aspects. Our findings show that the optimal batch size depends on both the dataset and the model. Selecting a batch size of one by default, which is often used for the classification tasks, does not always yield optimal results. Our findings highlight the importance of selecting a suitable batch size before performing classification tasks.