On the effectiveness of automated tracing from model changes to project issues

Abstract

Context: Requirements Traceability (RT) is concerned with monitoring and documenting the lifecycle of requirements. Although researchers have proposed several automated tracing tools, trace link establishment and maintenance are still prevalently manual activities.Objective: In order to foster the adoption of automated tracing tools, we study their empirical effectiveness in the context of model-driven development (MDD). We focus on trace link recovery (TLR) from (i) SVN revisions of MDD models to (ii) JIRA issues that represent requirements and bugs.Method: Based on the state-of-the-art in automated TLR, we propose the LCDTrace tool that uses 131 features to train a machine learning classifier. Some of these features use specific information for MDD contexts. We conduct three experiments on ten datasets from seven MDD projects. First, we evaluate the effectiveness of three ML algorithms and four rebalancing strategies using all 131 features, and we derive two optimal combinations for trace link recommendation and for trace maintenance. Second, we investigate whether the MDD-specific features convey higher performance than a version of LCDTrace that excludes those features. Third, we employ automated feature selection and study whether we can reduce the number of features while keeping similar performance, thereby boosting time and energy efficiency.Results: In our experiments, the gradient boosting models outperform those based on random forests. The best combinations for trace recommendation and maintenance achieve an F2-score of 61% and F0.5-score of 67%, respectively. While MDD-specific features do not provide additional value, automated feature selection succeeds at reducing feature numerosity without compromising performance.Conclusion: We provide insights on the effectiveness of state-of-the-art TLR techniques in MDD. Our findings are a baseline for devising and experimenting with alternative TLR approaches.