Skip to contents

Leave-one-out cross-validation (LOO-CV)

Source: R/fit.R
fit-method-loo.Rd

The $loo() method computes approximate LOO-CV using the loo package. In order to use this method you must compute and save the pointwise log-likelihood in your Stan program. See loo::loo.array() and the loo package vignettes for details.

Usage

loo(variables = "log_lik", r_eff = FALSE, moment_match = FALSE, ...)

Arguments

variables

(string) The name of the variable in the Stan program containing the pointwise log-likelihood. The default is to look for "log_lik". This argument is passed to the $draws() method.

r_eff

(multiple options) How to handle the r_eff argument for loo(). r_eff measures the amount of autocorrelation in MCMC draws, and is used to compute more accurate ESS and MCSE estimates for pointwise and total ELPDs.

  • TRUE will call loo::relative_eff.array() to compute the r_eff argument to pass to loo::loo.array().

  • FALSE (the default) or NULL will avoid computing r_eff, which can be very slow. The reported ESS and MCSE estimates may be over-optimistic if the posterior draws are far from independent.

  • If r_eff is anything else, that object will be passed as the r_eff argument to loo::loo.array().

moment_match

(logical) Whether to use a moment-matching correction for problematic observations. The default is FALSE. Using moment_match=TRUE will result in compiling the additional methods described in fit-method-init_model_methods. This allows CmdStanR to automatically supply the functions for the log_lik_i, unconstrain_pars, log_prob_upars, and log_lik_i_upars arguments to loo::loo_moment_match().

...

Other arguments (e.g., cores, save_psis, etc.) passed to loo::loo.array() or loo::loo_moment_match.default() (if moment_match = TRUE is set).

Value

The object returned by loo::loo.array() or loo::loo_moment_match.default().

References

  • Vehtari, A., Gelman, A., and Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413-1432. doi:10.1007/s11222-016-9696-4.

  • Vehtari, A., Simpson, D., Gelman, A., Yao, Y., and Gabry, J. (2024). Pareto smoothed importance sampling. Journal of Machine Learning Research, 25(72), 1-58.

  • Paananen, T., Piironen, J., Buerkner, P.-C., and Vehtari, A. (2021). Implicitly adaptive importance sampling. Statistics and Computing, 31, 16. doi:10.1007/s11222-020-09982-2 (for moment_match = TRUE).

See also

The loo package website with documentation and vignettes.

Examples


# \dontrun{
# the "logistic" example model has "log_lik" in generated quantities
fit <- cmdstanr_example("logistic")
loo_result <- fit$loo(cores = 2)
print(loo_result)
#> 
#> Computed from 4000 by 100 log-likelihood matrix.
#> 
#>          Estimate  SE
#> elpd_loo    -63.5 4.1
#> p_loo         3.8 0.5
#> looic       127.1 8.2
#> ------
#> MCSE of elpd_loo is 0.0.
#> MCSE and ESS estimates assume independent draws (r_eff=1).
#> 
#> All Pareto k estimates are good (k < 0.7).
#> See help('pareto-k-diagnostic') for details.
# }