Validating Factorial Survey Experiments: Response to Comment

Abstract

In Forster and Neugebauer (2024), we examine to what extent a factorial survey (FS) on invitations of fictitious applicants can replicate the findings of a nearly identical field experiment conducted with the same employers. In addition to exploring the conditions under which FSs provide valid behavioral predictions, we varied the topic sensitivity and tested whether behavioral predictions were more accurate after filtering out respondents who provided socially desirable answers or did not exert sufficient effort in responding to FS vignettes. Across these conditions, the FS results did not align well with the real-world benchmark. We conclude that researchers must exercise caution when using FSs to study (hiring) behavior. In this rejoinder, we respond to the critique of our study by Pickett (2025).