Estimating the impact of measurement differences introduced by efforts to reach a balanced response among non-western minorities

Abstract

This study investigates the impact of different modes and tailor-made response enhancing measures (TMREM)—such as bilingual interviewers with a shared ethnic background and translated questionnaires—on the measurement of substantive variables in surveys among minority ethnic groups in the Netherlands. It also provides insight into the ability to detect mode measurement effects of a recently developed method for disentangling mode measurement and mode selection effects, as well as into the tenability of the assumptions underlying this method. The data used in this study come from a large-scale survey design experiment among the four largest non-Western minority ethnic groups in the Netherlands comparing single-mode computer-assisted personal interviewing (CAPI) and sequential computer-assisted web interviewing, computer-assisted telephone interviewing, and CAPI-MM. The number and intensity of the TMREM varied among the four ethnic groups. The results show that mode measurement effects occur among all ethnic groups and are the result of a combination of the presence or absence of an interviewer and TMREM. Mode measurement effects occur more often on sociocultural questions, but also, on occasion, on more sociostructural or background questions. The method used to disentangle mode measurement and mode selection effects can be applied to detect mode measurement effects, but one should be cautious in interpreting them. Implausible mode measurement effects can be caused by the violation of the assumptions underlying this method.