Editorial: Validity, reliability and efficiency of comparative judgement to assess student work.

Abstract

Assessing complex skills such as writing, designing, or problem-solving is a challenge. Comparative judgement is considered to be a reliable and valid method for assessing student work (e.g., Lesterhuis et al., 2018; Verhavert et al., 2019). In comparative judgement, students’ work is evaluated by pairwise comparison. As assessors only must indicate which piece of work is better, differences in severity are not at play (Pollitt, 2012). Furthermore, each work is compared with several others and evaluated by multiple assessors. Based on these comparisons, the quality of each individual work can be estimated. This quality score reflects, so to speak, the shared consensus of the assessors (Jones et al., 2015; van Daal et al., 2019). The comparative judgement approach is based on Thurstone’s law of comparative judgement (1927), which states that it is possible to discriminate between objects on a single scale through a series of pairwise comparisons (Thurstone, 1927). Even though Thurstone already proposed the possibility of using comparative judgement for assessment in education, it was not until 2004 that Pollitt introduced the method in education in his paper “Let’s stop marking exams”. His work convincingly explained the merits of comparative assessment in terms of validity and provided the first evidence for a reliable summative assessment. Now, almost two decades later, various comparative judgement tools are available for education, such as Comproved or NoMoreMarking. Moreover, researchers around the world have investigated the quality of the method, where and/or how it can be applied, and how the method can be improved. In this Research Topic, we aim to provide a state-of-the-art of research on comparative judgement in education. We bring together current insights on the validity, reliability and efficiency of the method. In their contributions to this Research Topic, the authors present recent empirical research, each with their own approach, perspective and research focus. In this way, this Research Topic offers the foundation for future research into comparative judgement.