Can Survey Items Relevant to Mode-Specific Measurement Error Be Coded Reliably?: A Case Study on Eleven Official Dutch Surveys

Abstract

In multi-mode questionnaire design, usually some consideration is given to mode-specific measurement error. Despite this consideration, however, these measurement effects are frequently unexpectedly large. Measurement effects are determined by the interplay between characteristics of the questionnaire and characteristics of the respondents. For predicting measurement effects, we will first investigate the utility of so-called questionnaire profiles, in which item characteristics of questionnaires are summarized. As a first research question, we ask whether questionnaires can be coded reliably on item characteristics that are suggested in the literature as influential in evoking measurement effects. We constructed a typology of item characteristics from the literature and applied it to a wide range of national surveys. Results show that the characteristics difficult language usage, sensitive information, potential presumption of a filter question, emotional charge, and centrality could not be coded reliably. We will give possible explanations for these results and make suggestions for coping with low intercoder reliability.