SIMBA: A robust and generalizable measure of data imbalance

Abstract

Ranging from health to cybersecurity, real-world data are heavily imbalanced. Handling imbalance is among the formidable challenges of machine learning (ML), as it deteriorates ML’s performance, yielding biased results toward majority classes. However, finding an adequate measure to assess the impact of data imbalance is a field of research by itself. Following a review of the available imbalance measures, we introduce the status of imbalance (SIMBA), which considers data distribution and overlap, both of which are crucial to assess the impact of imbalance. SIMBA is benchmarked against seven imbalance measures on five ML models, 428 synthetic and 70 non-synthetic datasets from various domains. Resulting correlation coefficients between imbalance measures and classification performance and an analysis with 20 complexity measures prove that SIMBA consistently outperforms other measures. Overall, SIMBA accurately quantifies multiclass data imbalance and may help alleviate ML data imbalance challenges in the future.