• Introduction
• Overviews of Research Methods
• Journals
• Conceptual Issues
• Elements of the Posterior Distribution
• Bayesian Statistical Software
• Bayesian General and Generalized Latent Variable Models

# Bayesian StatisticsbyAlexander LoPilato, Mo WangLAST REVIEWED: 09 November 2020LAST MODIFIED: 27 July 2016DOI: 10.1093/obo/9780199846740-0102

## Introduction

Bayesian statistics refer to a general method of estimating statistical models. In contrast to classic or frequentist statistics, Bayesian statistics, also referred to as Bayesian methods, view the population parameter as a random variable instead of a fixed value. Bayesian methods are used to combine the information obtained from the observed data and the specified statistical model—in the form of the likelihood function—with the researchers’ prior beliefs about the effects under investigation (in the form of the prior distribution) to estimate a posterior distribution for each effect. The posterior distribution is a probability distribution that describes the uncertainty surrounding an effect. Because the posterior distribution is a combination of the likelihood function and the prior distribution, it can be changed by obtaining more data or changing the prior distribution to reflect different degrees of certainty about the effect. However, as the sample size increases, the results obtained from Bayesian methods converge to those obtained from frequentist methods. Bayesian methods are named after Reverend Thomas Bayes, who derived the Bayesian theorem. Using conditional probabilities, this theorem equates a model’s posterior distribution, which is a probability distribution for the model parameters conditional on the observed data, to the product of its likelihood function, which is informed only by the observed data, and the prior distribution, which is informed by a researcher’s prior beliefs about possible parameter values. The product is then rescaled or normalized by dividing it by the marginal probability of the observed data. Given this mathematical formulation, we can appreciate how the posterior distribution can be changed either by obtaining more data or by changing the prior distribution to reflect different degrees of certainty about the effect. However, as more data are collected and sample sizes increase, the “likelihood swamps the prior,” and results obtained from Bayesian methods converge to those obtained from frequentist methods. The advantage of Bayesian methods remains, however, because one still gets use language referring to the most probable parameters given the data.

## Overviews of Research Methods

Like frequentist methods, no text on Bayesian methods can cover all of the relevant statistical theories or the variety of modeling applications. However, there are many books that offer a good overview of Bayesian methods but vary in their accessibility. Perhaps the most well-known and most-used Bayesian text is Bayesian Data Analysis (Gelman, et al. 2013). This book offers a technical introduction to Bayesian methods that moves from the fundamentals of Bayesian inference all the way to fitting nonlinear and nonparametric Bayesian models. Cowles 2013, Gill 2014, as well as Song and Lee 2012 all offer strong, technical introductions to Bayesian methods. Cowles 2013 places more of an emphasis on modeling single and multiparameter distributions. Gill 2014 is written for social scientists and as such focuses on Bayesian hypothesis testing, estimating general linear models, and generalized hierarchical linear models. Song and Lee 2012 focuses on estimating basic and advanced structural equation models with Bayesian methods. Gelman and Hill 2007, Jackman 2009, Kaplan 2014, Kéry 2010, and Kruschke 2015 all offer introductions that are more accessible to non-quantitative social scientists. Gelman and Hill 2007 focuses on estimating Bayesian multilevel (hierarchical) linear models. Jackman 2009 offers a balanced view of statistical models commonly used by social scientists such as general linear models, multilevel models, and latent variable models. Similar to Jackman 2009, Kaplan 2014 focuses on statistical models commonly used in the social sciences but places more of an emphasis on the practical application of Bayesian models, not the theory behind them. Written for ecologists, Kéry 2010 offers an accessible introduction to Bayesian general and generalized linear models as well as their multilevel counterparts. Kruschke 2015 provides perhaps the most accessible and thorough introduction to Bayesian methods. Kruschke 2015 covers much of the same material as Kéry 2010 but in more mathematical detail. The BUGS Book (Lunn, et al. 2013) can function as an introduction to Bayesian methods, but it is perhaps best used as an introduction to the Bayesian estimation program, WinBUGS.

• Cowles, M. K. Applied Bayesian Statistics with R and OpenBUGS Examples. New York: Springer, 2013.

Provides a concise introduction to Bayesian statistics. It focuses on building the mathematical and computational foundations needed to estimate single and multiparameter probability models.

• Gelman, A., J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, and D. B. Rubin. Bayesian Data Analysis. 3d ed. New York: Chapman and Hall, 2013.

This is the authoritative text on Bayesian statistics. This book is broken into five different parts that cover the following: (1) the fundamentals of Bayesian inference, (2) the fundamentals of Bayesian data analysis, (3) advanced computation for Bayesian data analysis, (4) Bayesian regression models, and (5) Bayesian nonlinear and nonparametric models.

• Gelman, A., and J. Hill. Data Analysis Using Regression and Multilevel/Hierarchical Models. New York: Cambridge University Press, 2007.

Although the main purpose of this book is to introduce general and generalized linear models, as well as general and generalized linear mixed-effects models, it provides an introduction to how Bayesian methods can be used to estimate those models.

• Gill, J. Bayesian Methods: A Social and Behavioral Sciences Approach. 3d ed. Boca Raton, FL: Taylor and Francis, 2014.

Provides a technical introduction to Bayesian methods. It focuses heavily on the statistical foundations of Bayesian methods with several chapters on Markov Chain Monte Carlo theory/method.

• Jackman, S. Bayesian Analysis for the Social Sciences. Chichester, UK: Wiley, 2009.

This book provides a nice balance between Bayesian statistical theory and application. It devotes an entire chapter to Bayesian measurement theory and covers Bayesian factor analysis models and item response models.

• Kaplan, D. Bayesian Statistics for the Social Sciences. New York: Guilford, 2014.

Provides a comprehensive and accessible introduction to Bayesian methods. Its strength, however, is its coverage of Bayesian structural equation models including latent growth models, mixture models, and multilevel latent variable models.

• Kéry, M. Introduction to WinBUGS for Ecologists: A Bayesian Approach to Regression, ANOVA, Mixed Models, and Related Analyses. San Diego, CA: Elsevier, 2010.

Provides a relatively non-technical introduction to Bayesian methods. It briefly touches on the statistical foundations of Bayesian methods but then quickly moves on to its applications. One of the strengths of this book is that each chapter introduces a different statistical model and then compares the frequentist estimated model to the Bayesian estimated model.

• Kruschke, J. Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan. 2d ed. San Diego, CA: Elsevier, 2015.

Provides a nice mixture of Bayesian statistical theory and applications. It is a well-balanced, non-technical introduction to Bayesian methods. It is also one of the few texts to provide several chapters on Bayesian hypothesis testing and statistical power.

• Lunn, D., C. Jackson, N. Best, A. Thomas, and D. Spiegelhalter. The BUGS Book: A Practical Introduction to Bayesian Analysis. Boca Raton, FL: Taylor and Francis, 2013.

Although concise, this text provides a comprehensive overview of Bayesian methods. It was written by the researchers who created the widely used Bayesian analysis software: WinBUGS. It contains a useful list of WinBUGS commands and can function as a WinBUGS user guide.

• Song, X. -Y., and S. -Y. Lee. Basic and Advanced Bayesian Structural Equation Modeling with Applications in the Medical and Behavioral Sciences. Chichester, UK: Wiley, 2012.

A comprehensive overview of Bayesian structural equation modeling. Although technical, this book covers linear structural equation models (SEMs), nonlinear SEMs, multilevel SEMs, mixture SEMs, latent curve models, longitudinal SEMs, semi-parametric SEMs, and non-parametric SEMs.