The impact of summative, formative or programmatic assessment on the Dutch National Pharmacotherapy Assessment: a retrospective multicentre study

Abstract

BACKGROUND: The Dutch National Pharmacotherapy Assessment (DNPA), which focuses on assessing medication safety and essential drug knowledge, was introduced to improve clinical pharmacology and therapeutics education in the Netherlands. This study investigated how the performance of final-year medical students on the DPNA was affected by the assessment programme (traditional with summative or formative assessment, and programmatic assessment). METHODS: This multicentre retrospective longitudinal observation study (2019-2023) involved final-year medical students from four medical schools in the Netherlands. The DNPA was used in different ways - either as a summative or formative assessment in a traditional assessment programme or as a non-high-stakes assessment in a programmatic assessment programme. Three medical schools changed from assessment programme over time. RESULTS: This study involved 1894 students. Summative assessment resulted in significantly higher scores and pass rates than formative assessment in a traditional assessment programme (mean score of 84.3% vs. 67.5%, and pass rate of 60.4% vs. 5.9%). In contrast, slightly lower scores were obtained when the assessment was non-high-stakes as part of a programmatic assessment programme rather than a summative assessment in a traditional assessment programme (mean score of 81.% vs. 84.3%, pass rate of 51.8% vs. 60.4%). In curricula where the assessment became summative instead of formative, scores and pass rates significantly improved (mean increase of +14.4% and 42.3%, respectively), when the assessment programme changed from traditional with summative assessment to programmatic with non-high-stakes assessment, scores and pass rates modestly decreased (decrease of 3.3% and 14.2%, respectively). CONCLUSION: Integrating the DNPA within a traditional assessment programme is most effective when assessed summatively, as it results in significantly higher scores compared to formative assessment. In the context of a programmatic assessment programme, the scores may be slightly lower. Changing assessment programmes within a medical school influences DNPA scores. Scores increase when the assessment is summative rather than formative within a traditional assessment programme. Conversely, scores mildly decrease when the assessment programme shifts from traditional with summative assessment to non-high-stakes programmatic assessment.