Efficient Leave-One-Out Cross-Validation and WAIC for Bayesian Models • loo

loo

Efficient approximate leave-one-out cross-validation

loo is an R package that allows users to compute efficient approximate leave-one-out cross-validation for fitted Bayesian models, as well as model weights that can be used to average predictive distributions.

Getting Started

If you are just getting started with loo we recommend starting with the tutorial vignettes. There are also many examples throughout the package documentation.

Details

Leave-one-out cross-validation (LOO-CV, or LOO for short) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less used in practice because they involve additional computational steps.

The loo R package package implements the fast and stable computations for approximate LOO-CV and WAIC from

Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4. Online, arXiv preprint arXiv:1507.04544.
Vehtari, A., Gelman, A., and Gabry, J. (2017). Pareto smoothed importance sampling. arXiv preprint arXiv:1507.02646.

From existing posterior simulation draws, we compute approximate LOO-CV using Pareto smoothed importance sampling (PSIS), a new procedure for regularizing importance weights. As a byproduct of our calculations, we also obtain approximate standard errors for estimated predictive errors and for comparing predictive errors between two models.

We recommend PSIS-LOO-CV instead of WAIC, because PSIS provides useful diagnostics and effective sample size and Monte Carlo standard error estimates.

As of version 2.0.0, the loo package also provides methods for using stacking and other model weighting techiques to average Bayesian predictive distributions. For details on stacking and model weighting see:

Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2018). Using stacking to average Bayesian predictive distributions. In Bayesian Analysis, doi:10.1214/17-BA1091. Online, arXiv preprint arXiv:1704.02030.

Installation

Install the latest release from CRAN

install.packages("loo")

Install the latest development version from GitHub

if (!require(devtools)) {
  install.packages("devtools")
}
devtools::install_github("stan-dev/loo", build_vignettes = FALSE)

We do not recommend setting build_vignettes=TRUE when installing from GitHub because the vignettes take a long time to build and are always available online at mc-stan.org/loo/articles/.