Why is measuring Mathematical Knowledge for Teaching so hard? A struggle towards validation through student work analysis

Abstract

Measuring Mathematical Knowledge for Teaching (MKT) through open- response items proves challenging, with studies reporting variability in reliability and unclear subdomain boundaries. We illustrate this through the development of a 19-item test assessing three MKT subdomains - Specialized Content Knowledge (SCK), Knowledge of Content and Students (KCS) and Knowledge of Content and Teaching (KCT) - for Dutch pre-service teachers. Despite careful validation, the instrument demonstrated low internal consistency across subdomains (standardized =.36-.42). Examining two KCT items revealed two construct-dependent challenges. One item achieved high inter-rater reliability (ICC=.890) yet confounded SCK/KCT constructs, as indicated by qualitative analysis. The other item showed lower inter-rater reliability (ICC=.524); resolving this through strict conceptual scoring criteria revealed pre-service teachers predominantly provide procedural explanations, causing restricted variance and contributing to low internal consistency across the KCT subdomain. These findings suggest that achieving both theoretical purity and internal consistency may be difficult when participants demonstrate predominantly procedural knowledge.