Error Probabilities in Default Bayesian Hypothesis Testing

Abstract

This paper investigates the classical type I and type II error probabilities of default Bayes factors for a Bayesian tt test. Default Bayes factors quantify the relative evidence between the null hypothesis and the unrestricted alternative hypothesis without needing to specify prior distributions for the unknown parameters based on one’s prior beliefs. It is shown that in most typical situations in psychological research (when either observing no, small, medium or large effects) default Bayes factors are asymmetric in information, i.e., they result in unequal error probabilities. The tendency to either prefer the null hypothesis or the alternative hypothesis varies for different default Bayes factors. Although this asymmetry in information is a natural property of a Bayes factor, severe cases of asymmetry may be undesirable in a default setting because the underlying default priors are not a translation of one’s prior beliefs. A calibration scheme is presented to make a default Bayes factor symmetric in information under certain conditions.