`pareto-k-diagnostic.Rd`

Print a diagnostic table summarizing the estimated Pareto shape parameters
and PSIS effective sample sizes, find the indexes of observations for which
the estimated Pareto shape parameter \(k\) is larger than some
`threshold`

value, or plot observation indexes vs. diagnostic estimates.
The **Details** section below provides a brief overview of the
diagnostics, but we recommend consulting Vehtari, Gelman, and Gabry (2017a,
2017b) for full details.

pareto_k_table(x) pareto_k_ids(x, threshold = 0.5) pareto_k_values(x) psis_n_eff_values(x) mcse_loo(x, threshold = 0.7) # S3 method for psis_loo plot(x, diagnostic = c("k", "n_eff"), ..., label_points = FALSE, main = "PSIS diagnostic plot") # S3 method for psis plot(x, diagnostic = c("k", "n_eff"), ..., label_points = FALSE, main = "PSIS diagnostic plot")

x | |
---|---|

threshold | For |

diagnostic | For the |

label_points, ... | For the |

main | For the |

`pareto_k_table`

returns an object of class
`"pareto_k_table"`

, which is a matrix with columns `"Count"`

,
`"Proportion"`

, and `"Min. n_eff"`

, and has its own print method.

`pareto_k_ids`

returns an integer vector indicating which
observations have Pareto \(k\) estimates above `threshold`

.

`pareto_k_values`

returns a vector of the estimated Pareto
\(k\) parameters.

`psis_n_eff_values`

returns a vector of the estimated PSIS
effective sample sizes.

`mcse_loo`

returns the Monte Carlo standard error (MCSE)
estimate for PSIS-LOO. MCSE will be NA if any Pareto \(k\) values are
above `threshold`

.

The `plot`

method is called for its side effect and does not
return anything. If `x`

is the result of a call to `loo`

or `psis`

then `plot(x, diagnostic)`

produces a plot of
the estimates of the Pareto shape parameters (`diagnostic = "k"`

) or
estimates of the PSIS effective sample sizes (`diagnostic = "n_eff"`

).

The reliability and approximate convergence rate of the PSIS-based estimates can be assessed using the estimates for the shape parameter \(k\) of the generalized Pareto distribution:

If \(k < 0.5\) then the distribution of raw importance ratios has finite variance and the central limit theorem holds. However, as \(k\) approaches \(0.5\) the RMSE of plain importance sampling (IS) increases significantly while PSIS has lower RMSE.

If \(0.5 \leq k < 1\) then the variance of the raw importance ratios is infinite, but the mean exists. TIS and PSIS estimates have finite variance by accepting some bias. The convergence of the estimate is slower with increasing \(k\). If \(k\) is between 0.5 and approximately 0.7 then we observe practically useful convergence rates and Monte Carlo error estimates with PSIS (the bias of TIS increases faster than the bias of PSIS). If \(k > 0.7\) we observe impractical convergence rates and unreliable Monte Carlo error estimates.

If \(k \geq 1\) then neither the variance nor the mean of the raw importance ratios exists. The convergence rate is close to zero and bias can be large with practical sample sizes.

**If the estimated tail shape parameter \(k\) exceeds \(0.5\), the
user should be warned, although in practice we have observed good performance
for values of \(k\) up to 0.7.** (If \(k\) is greater than \(0.5\) then
WAIC is also likely to fail, but WAIC lacks its own diagnostic.)

If using PSIS in the context of approximate LOO-CV, even if the PSIS estimate has a finite variance the user should consider sampling directly from \(p(\theta^s | y_{-i})\) for any problematic observations \(i\), use \(K\)-fold cross-validation, or use a more robust model. Importance sampling is likely to work less well if the marginal posterior \(p(\theta^s | y)\) and LOO posterior \(p(\theta^s | y_{-i})\) are much different, which is more likely to happen with a non-robust model and highly influential observations. A robust model may reduce the sensitivity to highly influential observations.

Vehtari, A., Gelman, A., and Gabry, J. (2017a). Practical
Bayesian model evaluation using leave-one-out cross-validation and WAIC.
*Statistics and Computing*. 27(5), 1413--1432.
doi:10.1007/s11222-016-9696-4.
(
journal, preprint arXiv:1507.04544).

Vehtari, A., Gelman, A., and Gabry, J. (2017b). Pareto smoothed importance sampling. arXiv preprint: http://arxiv.org/abs/1507.02646/

`psis`

for the implementation of the PSIS algorithm.