Crowd-Based Requirements Engineering

Abstract

The high availability of data on the Internet has opened up new avenues for software development and deployment. For Requirements Engineering (RE), in particular, online user feedback can provide valuable insights into the behavior and opinions of stakeholders, enabling novel types of analysis. To help RE integrate techniques for analyzing feedback from a large group of stakeholders—a crowd—in fast and iterative cycles, the research of this PhD dissertation introduces the Crowd-based Requirements Engineering (CrowdRE) paradigm. This describes a semi-automated RE approach for obtaining and analyzing user feedback from a crowd, with the goal of deriving validated user requirements. The associated CrowdRE framework recommends the integration of text and usage mining results and continuous engagement of the crowd. This research effort is captured in the following main research question, which we answer in three parts: How can software development departments employ the crowd to improve their products? First, we define and scope CrowdRE. This requires defining a paradigm in terms of the actors, approaches, and resources it should comprise. We demonstrate that there has been no systematic integration of relevant RE techniques within a holistic framework before, provide definitions for CrowdRE and crowd, create a user feedback taxonomy, and position CrowdRE within the existing RE landscape to justify the introduction of a novel paradigm, which identifies four key activities for CrowdRE: (1) motivating crowd members, (2) eliciting user feedback, (3) analyzing user feedback, and (4) monitoring context and usage data. We also consider whether CrowdRE is compliant with the principles of the EU General Data Protection Regulation (GDPR). We then explore how analyzing online user feedback can contribute to gaining a deep understanding of user requirements. The applicability of CrowdRE depends on whether user feedback contains requirements-relevant information and how these insights can be used for software development projects. We demonstrate that online user feedback contains expressions about nearly all quality characteristics and subcharacteristics of the leading ISO 25010 standard’s software product quality taxonomy. This suggests that, in addition to identifying features, CrowdRE can support the often difficult task of eliciting quality requirements. Using a fictitious case, we also show how CrowdRE can be applied continuously and how it can augment existing RE and crowdsourcing activities. Lastly, we design CrowdRE solutions that employ automation to identify quality aspects in online user feedback efficiently and effectively. The lack of large training corpora prevents machine learning tools from producing reliable results. This is why we construct and evaluate three approaches: (1) linguistic patterns using an adaptation of the NFR Method, (2) crowdsourcing using our Kyōryoku method, and (3) a state-of-the-art large language model (LLM) pipeline. The best-performing configurations for crowdsourcing and LLMs were equally accurate in classifying quality aspects in online user feedback, both performing considerably better than the linguistic patterns. CrowdRE has become an established subfield of RE with a growing collection of techniques that software development departments can use to obtain insights into the user perspective on their products, based on which they can make continuous improvements.