Bayesian Hypothesis Testing Illustrated: An Introduction for Software Engineering Researchers

Bayesian data analysis is gaining traction in many fields, including empirical studies in software engineering. Bayesian approaches provide many advantages over traditional, or frequentist, data analysis, but the mechanics often remain opaque to beginners due to the underlying computational complexity. Introductory articles, while successful in explaining the theory and principles, fail to provide a totally transparent operationalization. To address this gap, this tutorial provides a step-by-step illustration of Bayesian hypothesis testing in the context of software engineering research using a fully developed example and in comparison to the frequentist hypothesis testing approach. It shows how Bayesian analysis can help build evidence over time incrementally through a family of experiments. It also discusses chief advantages and disadvantages in an applied manner. A figshare package is provided for reproducing all calculations.