Testing replication with small samples: Applications to ANOVA

Abstract

Findings based on small samples can offer important insights, but original small sample findings should be replicated before strong conclusions can be drawn. In this chapter, we present four common replication research questions: (1) whether the new effect size is similar to the original effect size, (2) whether the new effect size differs from the original effect size, (3) whether the conclusions based on new results differ from the original conclusions, and (4) what the effect size is in the population. For each of these research questions, we discuss appropriate evaluation methods: replication Bayes factors, confidence intervals, prediction intervals, the prior predictive p-value, and bias-corrected meta-analysis methods. Each method is illustrated for the replication of an ANOVA and associated post hoc t-test. Annotated R-code for all analyses is provided with the chapter.