Misuse of the sign test in narrative synthesis of evidence

Abstract

In narrative synthesis of evidence, it can be the case that the only quantitative measures available concerning the efficacy of an intervention is the direction of the effect, that is, whether it is positive or negative. In such situations, the sign test has been proposed in the literature and in recent Cochrane guidelines as a way to test whether the proportion of positive effects is favorable. I argue that the sign test is inappropriate in this context as the data are not generated according to the binomial distribution it employs. I demonstrate possible consequences for both hypothesis testing and estimation via hypothetical examples.