Sentiment analysis for software quality assessment

Abstract

During the software selection process, software engineers often rely on text reviews from repository platforms, communities, or forums to collect software quality information. As these reviews offer direct insight into users' experience and perception of the software components. However, text reviews are often formulated implicitly, and the process of gathering user feedback from multiple sources can be a time-intensive endeavor, posing challenges in the collection and analysis of substantial volumes of data. We conducted a systematic literature review to explore the state-of-the-art solutions in sentiment analysis. By leveraging the knowledge derived from the literature review, we developed a sentiment analysis tool to measure software component quality by analyzing the sentiment of reviews from experienced users. Our goal is to provide a channel to help software stakeholders gain insight into the software quality attributes, thus enhancing the overall health of software and the software ecosystem. This tool consists of TextRank, which extracts keywords related to software quality attributes from raw data, an Aho-Corasick automaton used to search for these keywords in reviews and map them to software quality attributes, and a sentiment analysis model to perform sentiment analysis. We compare four widely mentioned models in the literature review, namely BERT, BERT-BiLSTM, BERT-BiLSTM-Attention, and RoBERTa, in terms of performance metrics such as accuracy, F1, precision, and recall. BERT-BiLSTM-Attention is selected as the sentiment analysis model due to its superior performance in both training and test datasets. In addition, we integrated a decision algorithm that computes the fuzzy group consensus sentiment for the relevant quality attributes of each software component and visualizes it through a sentiment quality matrix.