The relative size of measurement error and attrition error in a panel survey. Comparing them with a new multi-trait multi-method model

Abstract

This paper proposes a method to simultaneously estimate both measurement and nonresponse errors for attitudinal and behavioural questions in a longitudinal survey. The method uses a Multi-Trait Multi-Method (MTMM) approach, which is commonly used to estimate the reliability and validity of survey questions. The classic MTMM model is in this paper extended to include the effects of measurement bias and longitudinal nonresponse that occurs in longitudinal surveys. Measurement and nonresponse errors are expressed on a common metric in this model, so that their relative sizes can be assessed over the course of a panel study. Using an example about political trust from the Dutch LISS panel, we show that measurement problems lead to both small errors and small biases, that dropout in the panel study does not lead to errors or bias, and that therefore, measurement is a more important source of both error and bias than nonresponse.