Why is measuring Mathematical Knowledge for Teaching so hard? A struggle towards validation through student work analysis

Measuring Mathematical Knowledge for Teaching (MKT) through open-
response items proves challenging, with studies reporting variability in
reliability and unclear subdomain boundaries. We illustrate this through the
development of a 19-item test assessing three MKT subdomains -
Specialized Content Knowledge (SCK), Knowledge of Content and
Students (KCS) and Knowledge of Content and Teaching (KCT) - for
Dutch pre-service teachers. Despite careful validation, the instrument
demonstrated low internal consistency across subdomains (standardized
=.36-.42). Examining two KCT items revealed two construct-dependent
challenges. One item achieved high inter-rater reliability (ICC=.890) yet
confounded SCK/KCT constructs, as indicated by qualitative analysis. The
other item showed lower inter-rater reliability (ICC=.524); resolving this
through strict conceptual scoring criteria revealed pre-service teachers
predominantly provide procedural explanations, causing restricted variance
and contributing to low internal consistency across the KCT subdomain.
These findings suggest that achieving both theoretical purity and internal
consistency may be difficult when participants demonstrate predominantly
procedural knowledge.