On Comparative Statistics for Labelling Tasks: What can We Learn from MIREX ACE 2013

Abstract

For mirex 2013, the evaluation of audio chord estimation (ace) followed a new scheme. Using chord vocabularies of differing complexity as well as segmentation measures, the new scheme provides more information than the ace evaluations from previous years. With this new information, however, comes new interpretive challenges. What are the correlations among different songs and, more importantly, different submissions across the new measures? Performance falls off for all submissions as the vocabularies increase in complexity, but does it do so directly in proportion to the number of more complex chords, or are certain algorithms indeed more robust? What are the outliers, songalgorithm pairs where the performance was substantially higher or lower than would be predicted, and how can they be explained? Answering these questions requires moving beyond the Friedman tests that have most often been used to compare algorithms to a richer underlying model. We propose a logistic-regression approach for generating comparative statistics for mirex ace, supported with generalised estimating equations (gees) to correct for repeated measures. We use the mirex 2013 ace results as a case study to illustrate our proposed method, including some of interesting aspects of the evaluation that might not apparent from the headline results alone.