Guidelines for the validation of writing assessment in intervention studies

Abstract

Valid conclusions about the effectiveness of writing interventions depend for a large part on how students’ writing proficiency is assessed. Researchers have to make many choices when designing a writing assessment, all of which may impact a valid interpretation and use of the assessment scores. This chapter aims to support researchers in thinking through the inferences and assumptions underlying these choices and the extent to which they can ultimately claim a valid interpretation of the effectiveness of the intervention on the basis of students’ writing assessment performance. To that aim we will use Kane’s argument-based approach to validity and apply it to the context of writing intervention research. This framework discerns five inferences, or steps, in the claim from the observed writing performance to decisions on the effectiveness of the intervention: selecting (writing) tasks that represent the central construct of writing in the intervention, scoring students’ performance on the writing tasks, generalizing scores to the defined writing construct, extrapolating scores to the broader domain of writing, and making decisions about the effectiveness of the intervention. To support researchers in justifying each of these inferences, we bring together existing research on writing interventions and writing assessment. This results in specific guidelines for researchers on how to make evidence-based choices when designing the writing assessment in their intervention study, and hence, to support the validity of their decisions about the effects of the intervention.