Unraveling programmatic assessment: Issues of validity in high-stakes decision-making

Abstract

Within health professions programs programmatic assessment has been proposed to optimize the assessment of competency-based education. Programmatic assessment aims to create a coherent system of assessment. By providing meaningful feedback, together with creating follow-up, the learner has the opportunity to learn and improve its performance. Pass/fail decisions (i.e., high-stakes decisions) do not take place after each assessment. Alternatively, every assessment is low-stakes and only after a longer period of time a high-stakes decision is made. In this thesis we have tried to unravel programmatic assessment by investigating whether validity evidence can be provided for high-stakes decision-making. Can we justify the high-stakes decision about the learner’s progression and performance within a programmatic approach to assessment? To investigate whether it is possible to generate sufficient and reliable information about the learner as input for the high-stakes decision, we performed a quantitative analysis of the scores provided by raters over time. Furthermore, not only the raters are essential in creating the information to inform the high-stakes decision, learners are involved as well by seeking feedback. To learn more about this, we investigated learner’s motivation toward feedback-seeking. Next to that, the concept of saturation of information is important. We did an initial exploration of the concept of saturation of information by investigating the relationship between the number of datapoints and the inter-rater agreement of the examiners involved in high-stakes decision-making. Then, we dived deeper into the concept by investigating the content of the datapoints, i.e., the quality of narrative information. In the final part of this thesis we investigated whether, and if so, what differences exist in designing a high-stakes decision-making procedure between programs within health professions education. As a follow-up on this, in a commentary we proposed an idea to take a different view on the role of the committee consisting of examiners performing the high-stakes decision. As the final chapter, we discussed the results of the studies performed in this thesis within existing literature and propose suggestions for further research.