Analysis Blinding as a Potential Means to Foster a Productive Collaboration Between Original Authors and Replicators

Abstract

Recent awareness of the importance of rigor and robustness have deemed replication efforts vital for scientific advance. Yet the value of replication projects may often be undermined by post-hoc disputes with the original authors about the replication outcomes, for instance, concerning data quality, unanticipated deviations from the data collection protocol, or diverging implementations of the analysis strategy. In this comment, we reflect on the tension between replicators and original authors and advocate for analysis blinding as a means to prevent such unproductive post-hoc discussions. Analysis blinding involves the alteration of data to remove the key effect of interest while preserving all other aspects. This methodology allows for an assessment of important properties of the data (manipulation checks, outliers, data quality) without introducing bias or risking the perception of attempting to manipulate the results. We discuss three replication studies we were responsible for in the Holzmeister et al. (2025) project and demonstrate how to effectively blind data for each of them. We argue that analysis blinding has the potential to prevent fruitless discussions and tension between original authors and replication teams in replication projects while preserving a healthy scientific debate.