Introduction
This vignette shows how to use the latent projection predictive feature selection from Catalina et al. (2021) in projpred. We recommend to read the main vignette first, as the latent-projection vignette presented here will skip some of the details explained in the main vignette.
General idea
The response families used in GLMs (McCullagh and Nelder 1989, chap.
2), GLMMs, GAMs, and GAMMs (in particular, the
gaussian(), the binomial(), and the
poisson() family which are supported by
projpred’s traditional projection) may be termed
exponential dispersion (ED) families (Jørgensen 1987)1. For a
response family that is not an ED family, the Kullback-Leibler (KL)
divergence minimization problem (see Piironen et al. 2020) is
often not easy to solve analytically (exceptions are non-ED families
that are discrete and have finite support; see the comment on the
augmented-data projection in section “Implementation”). In order to bypass this issue, the
latent projection (Catalina et al. 2021) solves
the KL minimization problem in the predictive space of the latent
predictors2 instead of in the predictive space of the
original response values.
To this end, the latent predictor is assumed to have a Gaussian distribution, since it (i) constitutes a combination of predictor data and regression parameters which is often linear (in the parameters, but—less—often also in the predictor data) or at least additive (across the predictor terms) and (ii) has the complete real line as support. Furthermore, the Gaussian distribution has the highest differential entropy among all distributions with two finite moments and with the real line as support (see, e.g., Cover and Thomas 1991). In some cases, e.g., for the probit link, the Gaussian distribution is even part of the original statistical model. In case of the logit link, the Gaussian distribution with a standard deviation of 1.6 approximates the logistic distribution (with a scale parameter of 1).
The assumption of a Gaussian distribution for the latent predictors makes things a lot easier because it allows us to make use of projpred’s traditional projection.
As illustrated by the Poisson example below, the latent projection can not only be used for families not supported by projpred’s traditional projection, but it can also be beneficial for families supported by it. The remaining examples show that the latent projection offers support for models (and censored observations) that the traditional (or augmented-data) projection do not support.
Implementation
To use the latent projection in projpred, argument
latent of extend_family() needs to be set to
TRUE. Since extend_family() is called by
init_refmodel() which in turn is called by
get_refmodel() (more precisely, by the
get_refmodel() methods) which in turn is called at the
beginning of the top-level functions project(),
varsel(), and cv_varsel(), it is possible to
pass latent = TRUE from such a top-level function down to
extend_family() via the ellipsis (...).
However, we recommend to define the reference model object of class
refmodel explicitly (as illustrated in the examples below)
to avoid repetitive and inefficient code3.
After performing the projection (either as a stand-alone feature via
project() or embedded in a variable selection via
varsel() or cv_varsel()), the post-processing
(e.g., the estimation of the performance statistics in
summary.vsel()) can be performed on the original response
scale. For this, there are three arguments of
extend_family() which accept R functions:
latent_ilink (responsible for the inverse-link
transformation from latent scale to response scale),
latent_ll_oscale (responsible for the calculation of
log-likelihood values on response scale), and
latent_ppd_oscale (responsible for drawing from the
(posterior-projection) predictive distribution on response scale). For
some families, these three arguments have internal defaults implemented
natively in projpred. These families are listed in the
main vignette (section “Supported
types of models”). For all other families, projpred
either tries to infer a reasonable function internally (in case of
latent_ilink) or uses a dummy function returning only
NAs (in case of latent_ll_oscale and
latent_ppd_oscale), unless the user supplies custom
functions. When creating a reference model object for a family that
lacks projpred’s native support for full response-scale
post-processing, projpred will throw messages stating
whether (and which) features will be unavailable unless at least some of
these three arguments are provided by the user. Again, the ellipsis
(...) can be used to pass these arguments from a top-level
function such as cv_varsel() down to
extend_family(). In the post-processing functions,
response-scale analyses can usually be deactivated by setting argument
resp_oscale to FALSE, with the exception of
predict.refmodel() and proj_linpred() where
arguments type and transform serve this
purpose (see the documentation).
Apart from the arguments mentioned above,
extend_family() also features the latent-projection
argument latent_y_unqs whose purpose is described in the
documentation.
While the latent projection is an approximate solution to the KL divergence minimization problem in the original response space4, the augmented-data projection (Weber et al. 2025) gives the exact5 solution for some non-ED families, namely those where the response distribution has finite support. However, the augmented-data projection comes with a higher runtime than the latent projection. The families currently supported by projpred’s augmented-data projection are also listed in the main vignette (again section “Supported types of models”).
Example: Poisson distribution
In this example, we will illustrate that in case of a family
supported by projpred’s traditional projection (here
the poisson() family), the latent projection can reduce the
runtime of the variable selection compared to the traditional
projection.
Data
First, we generate a dataset with a Poisson-distributed response. The linear predictor depends on 4 continuous predictors and 1 categorical predictor, and there are 20 noise predictors:
# Number of observations:
N <- 100
# Data-generating function:
sim_poiss <- function(nobs = N, ncon = 4, ncats = 4, nnoise = 20) {
# Regression coefficients for continuous predictors:
coefs_con <- seq(from = 0.2, to = 0.8, length.out = ncon)
# Continuous predictors:
dat_sim <- matrix(rnorm(nobs * ncon), ncol = ncon)
# Start linear predictor:
linpred <- 1.4 + dat_sim %*% coefs_con
# Categorical predictor:
dat_sim <- data.frame(
x = dat_sim,
xcat = gl(n = ncats, k = nobs %/% ncats, length = nobs,
labels = paste0("cat", seq_len(ncats)))
)
# Regression coefficients for the categorical predictor:
coefs_cat <- seq(from = 0.2, to = 0.8, length.out = ncats)
# Continue linear predictor:
linpred <- linpred + coefs_cat[dat_sim$xcat]
# Noise predictors:
dat_sim <- data.frame(
dat_sim,
xn = matrix(rnorm(nobs * nnoise), ncol = nnoise)
)
# Poisson response, using the log link (i.e., exp() as inverse link):
dat_sim$y <- rpois(nobs, lambda = exp(linpred))
# Shuffle order of observations:
dat_sim <- dat_sim[sample.int(nobs), , drop = FALSE]
# Drop the shuffled original row names:
rownames(dat_sim) <- NULL
return(dat_sim)
}
# Generate data:
set.seed(300417)
dat_poiss <- sim_poiss()Reference model
Next, we fit the reference model that we consider as the best model (in terms of predictive performance) that we can construct (here, we assume that we don’t know about the true data-generating process even though the dataset was simulated):
[1] 25
# Prior guess for the number of relevant (i.e., non-zero) regression
# coefficients:
p0 <- 5
# Prior guess for the overall magnitude of the response values, see Table 1 of
# Piironen and Vehtari (2017, DOI: 10.1214/17-EJS1337SI):
mu_prior <- 10
# Hyperprior scale for tau, the global shrinkage parameter:
tau0 <- p0 / (D - p0) / sqrt(mu_prior) / sqrt(N)
# Set this manually if desired:
ncores <- parallel::detectCores(logical = FALSE)
### Only for technical reasons in this vignette (you can omit this when running
### the code yourself):
ncores <- min(ncores, 2L)
###
options(mc.cores = ncores)
# All other columns of the data are predictors:
refm_fml <- y ~ .
refm_fit_poiss <- stan_glm(
formula = refm_fml,
family = poisson(),
data = dat_poiss,
prior = hs(global_scale = tau0, slab_df = 100, slab_scale = 1),
### Only for the sake of speed (not recommended in general):
chains = 2, iter = 1000,
### In general, use defaults or explicitly define
### chains = 4, iter = 4000,
###
refresh = 0
)Variable selection using the latent projection
Within projpred, we define the reference model
object explicitly and set latent = TRUE in the
corresponding get_refmodel() call (see section “Implementation”) so that the latent projection is used
in downstream functions. We run cv_varsel() with its
defaults: PSIS-LOO CV and validated search. Furthermore, we measure the
runtime to be able to compare it to the traditional projection’s
later:
refm_poiss <- get_refmodel(refm_fit_poiss, latent = TRUE)
time_lat <- system.time(cvvs_lat <- cv_varsel(
refm_poiss,
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0,
###
seed = 95930
))Cutting off the search at size 19. Use argument `nterms_max` to customize this.
Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (6 / 100) Pareto k's for the reference model's PSIS-LOO weights
are > 0.67.
Warning: In the recalculation of the latent response values, some (6 / 100) expectation-specific Pareto k-values are > 0.67.
In general, we recommend K-fold CV in this case.
print(time_lat) user system elapsed
17.843 0.010 4.789
The warnings about high Pareto \(\hat{k}\) values concern only a few of the
100 observations. With the short MCMC run of this vignette (1000
posterior draws), the threshold for a reliable Pareto \(\hat{k}\) is about 0.67 (Vehtari et al.
2022). In a real analysis, we would run more MCMC iterations,
check the Pareto \(\hat{k}\) values
again, and use \(K\)-fold CV
(cv_method = "kfold") if many of them are high.
In order to decide for a submodel size, we first inspect the
plot() results. In contrast to the main vignette where we
used the mean log predictive density (MLPD) as predictive performance
statistic for a gaussian() family reference model (and
gaussian() submodels), we have a discrete family
(poisson()) here, so it makes sense to exponentiate the
MLPD to obtain the geometric mean predictive density (GMPD; in case of a
discrete response, the predictive density values are actually predictive
probabilities and hence the GMPD is bounded by 0 and 1). As in
the main vignette, we plot with deltas = TRUE (in case of
the GMPD, this means that the ratio of the submodel GMPD
vs. the reference model GMPD is shown). Via global option
projpred.plot_vsel_size_position, we set argument
size_position of plot.vsel() to
"secondary_x" to make the submodel sizes readable in all of
the plots in this vignette.
ggplot2::theme_set(bayesplot::theme_default(base_family = "sans"))
options(projpred.plot_vsel_size_position = "secondary_x")
( gg_lat <- plot(cvvs_lat, stats = "gmpd", deltas = TRUE) )
Based on this plot, we decide for a submodel size of 5:
size_decided_lat <- 5This is also the size that suggest_size() would
suggest:
suggest_size(cvvs_lat, stat = "gmpd")[1] 5
In the predictor ranking up to the selected size of 5, we can see that projpred has correctly selected the truly relevant predictors first and only then the noise predictors:
rk_lat <- ranking(cvvs_lat)
( predictors_final_lat <- head(rk_lat[["fulldata"]], size_decided_lat) )[1] "x.4" "x.3" "x.2" "xcat" "x.1"
We will skip post-selection inference here (see the main vignette for
a demonstration of post-selection inference), but note that
proj_predict() has argument resp_oscale for
controlling whether to draw from the posterior-projection predictive
distributions on the original response scale (TRUE, the
default) or on latent scale (FALSE) and that analogous
functionality is available in proj_linpred() (argument
transform) and predict.refmodel() (argument
type).
Variable selection using the traditional projection
We now run the same cv_varsel() call with
projpred’s traditional projection. For a Poisson
submodel, each projection is an iterative fit, and validated search
repeats the search for each of the 100 LOO CV folds. To keep the runtime
of this vignette short, we use fewer clusters in the performance
evaluation (nclusters_pred = 20) and search only up to size
10 (nterms_max = 10):
time_trad <- system.time(cvvs_trad <- cv_varsel(
refm_fit_poiss,
### Only for the sake of speed (not recommended in general):
nclusters_pred = 20,
nterms_max = 10,
###
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0,
###
seed = 95930
))Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (6 / 100) Pareto k's for the reference model's PSIS-LOO weights
are > 0.67.
rbind(latent = time_lat, traditional = time_trad)[, "elapsed"] latent traditional
4.789 59.002
suggest_size(cvvs_trad, stat = "gmpd")[1] 5
[1] "x.4" "x.3" "x.2" "xcat" "x.1"
( gg_trad <- plot(cvvs_trad, stats = "gmpd", deltas = TRUE) )
The traditional projection takes much longer than the latent projection (see the elapsed times in seconds above), although it uses the smaller settings. It selects the same 5 predictor terms first.
Example: Negative binomial distribution
In this example, we will illustrate the latent projection in case of
the negative binomial family (more precisely, we will use the
rstanarm::neg_binomial_2() family here) which is a family
that is not supported by projpred’s traditional
projection6.
Reference model
We now fit a reference model with the negative binomial distribution
as response family. For the sake of simplicity, we won’t adjust
tau0 to this new family, but in a real-world example, such
an adjustment would be necessary. However, since Table 1 of Piironen and Vehtari (2017) does not list the
negative binomial distribution, this would first require a manual
derivation of the pseudo-variance \(\tilde{\sigma}^2\).
refm_fit_nebin <- stan_glm(
formula = refm_fml,
family = neg_binomial_2(),
data = dat_poiss,
prior = hs(global_scale = tau0, slab_df = 100, slab_scale = 1),
### Only for the sake of speed (not recommended in general):
chains = 2, iter = 1000,
###
refresh = 0
)If a warning about a low effective sample size (ESS) appears, we
ignore it here because of the technical reasons for which we reduced
chains and iter in this vignette.
Variable selection using the latent projection
To request the latent projection with latent = TRUE, we
now need to specify more arguments (latent_ll_oscale and
latent_ppd_oscale; the internal default for
latent_ilink works correctly in this example) which will be
passed to extend_family()7:
refm_prec <- as.matrix(refm_fit_nebin)[, "reciprocal_dispersion", drop = FALSE]
latent_ll_oscale_nebin <- function(ilpreds,
dis = rep(NA, nrow(ilpreds)),
y_oscale,
wobs = rep(1, ncol(ilpreds)),
cens,
cl_ref,
wdraws_ref = rep(1, length(cl_ref))) {
y_oscale_mat <- matrix(y_oscale, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
wobs_mat <- matrix(wobs, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
refm_prec_agg <- cl_agg(refm_prec, cl = cl_ref, wdraws = wdraws_ref)
ll_unw <- dnbinom(y_oscale_mat, size = refm_prec_agg, mu = ilpreds, log = TRUE)
return(wobs_mat * ll_unw)
}
latent_ppd_oscale_nebin <- function(ilpreds_resamp,
dis_resamp = rep(NA, nrow(ilpreds_resamp)),
wobs = rep(1, ncol(ilpreds_resamp)),
cl_ref,
wdraws_ref = rep(1, length(cl_ref)),
idxs_prjdraws) {
refm_prec_agg <- cl_agg(refm_prec, cl = cl_ref, wdraws = wdraws_ref)
refm_prec_agg_resamp <- refm_prec_agg[idxs_prjdraws, , drop = FALSE]
ppd <- rnbinom(prod(dim(ilpreds_resamp)), size = refm_prec_agg_resamp,
mu = ilpreds_resamp)
ppd <- matrix(ppd, nrow = nrow(ilpreds_resamp), ncol = ncol(ilpreds_resamp))
return(ppd)
}
refm_nebin <- get_refmodel(refm_fit_nebin, latent = TRUE,
latent_ll_oscale = latent_ll_oscale_nebin,
latent_ppd_oscale = latent_ppd_oscale_nebin)
cvvs_nebin <- cv_varsel(
refm_nebin,
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0
###
)Cutting off the search at size 19. Use argument `nterms_max` to customize this.
Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (2 / 100) Pareto k's for the reference model's PSIS-LOO weights
are > 0.67.
Warning: In the recalculation of the latent response values, some (2 / 100) expectation-specific Pareto k-values are > 0.67.
In general, we recommend K-fold CV in this case.
Again, we first inspect the plot() results to decide for
a submodel size:
( gg_nebin <- plot(cvvs_nebin, stats = "gmpd", deltas = TRUE) )
Based on this plot, we decide for a submodel size of 5:
size_decided_nebin <- 5This is also the size that suggest_size() would
suggest:
suggest_size(cvvs_nebin, stat = "gmpd")[1] 5
As we can see from the predictor ranking included in the plot, our selected 5 predictor terms are exactly the truly relevant ones. More explicitly, our selected predictor terms are:
rk_nebin <- ranking(cvvs_nebin)
( predictors_final_nebin <- head(rk_nebin[["fulldata"]],
size_decided_nebin) )[1] "x.4" "x.3" "x.2" "xcat" "x.1"
Again, we will skip post-selection inference here (see the main vignette for a demonstration of post-selection inference).
Censored observations (survival analysis)
Response-scale analyses for models with censored responses (e.g., for
time-to-event models as occurring in survival analysis) are possible
when using the latent projection in combination with a custom
latent_ll_oscale function (see ?extend_family)
that has an attribute called cens_var and makes use of its
argument cens8.
We will illustrate this here using simulated data and right censoring, first with a Weibull response and then with a log-normal response. Both examples are adapted from a thread on The Stan Forums. Here, we use the following common simulation part:
N_surv <- 500
n_pred <- 50
n_pred_truth <- 10
n_pred_noise <- n_pred - n_pred_truth
dat_sim_surv <- matrix(rnorm(N_surv * n_pred), ncol = n_pred)
colnames(dat_sim_surv) <- paste0("x", c(paste0(".", seq_len(n_pred_truth)),
paste0("n.", seq_len(n_pred_noise))))
linpreds_surv <- -0.1 +
dat_sim_surv[, seq_len(n_pred_truth), drop = FALSE] %*%
rep_len(c(0.3, -0.2), length.out = n_pred_truth)
epreds_surv <- exp(linpreds_surv)
dat_sim_surv <- as.data.frame(dat_sim_surv)
cens_surv <- runif(N_surv,
min = quantile(epreds_surv, probs = 0.4),
max = quantile(epreds_surv, probs = 0.9))Example: Weibull distribution with right-censored observations
For the Weibull model, we complete our data generation as follows:
shape_weib <- 1.2
scales_weib <- epreds_surv / (gamma(1 + (1 / shape_weib)))
y_weib <- rweibull(N_surv, shape = shape_weib, scale = scales_weib)
is_event_weib <- y_weib < cens_surv
yobs_weib <- y_weib
yobs_weib[!is_event_weib] <- cens_surv[!is_event_weib]
dat_sim_weib <- data.frame(yobs = yobs_weib,
is_censored = 1 - is_event_weib,
dat_sim_surv)We now fit a reference model in brms:
refm_fit_weib <- brms::brm(
formula = yobs | cens(is_censored) ~ .,
family = brms::weibull(),
data = dat_sim_weib,
prior = brms::prior(R2D2(mean_R2 = 1/3, prec_R2 = 3)),
### Only for the sake of speed (not recommended in general):
chains = 2,
###
silent = 2,
refresh = 0
)The following code prepares the cv_varsel() run and the
downstream projpred steps:
refm_shape <- as.matrix(refm_fit_weib)[, "shape", drop = FALSE]
latent_ll_oscale_weib <- structure(function(
ilpreds,
dis = rep(NA, nrow(ilpreds)),
y_oscale,
wobs = rep(1, ncol(ilpreds)),
cens,
cl_ref,
wdraws_ref = rep(1, length(cl_ref))
) {
idxs_cens <- which(cens == 1)
idxs_event <- setdiff(seq_along(cens), idxs_cens)
wobs_mat <- matrix(wobs, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
refm_shape_agg <- cl_agg(refm_shape, cl = cl_ref, wdraws = wdraws_ref)
ll_unw <- matrix(nrow = nrow(ilpreds), ncol = ncol(ilpreds))
for (idx_cens in idxs_cens) {
ll_unw[, idx_cens] <- pweibull(
y_oscale[idx_cens],
shape = refm_shape_agg,
scale = ilpreds[, idx_cens] / gamma(1 + 1 / as.vector(refm_shape_agg)),
lower.tail = FALSE,
log.p = TRUE
)
}
for (idx_event in idxs_event) {
ll_unw[, idx_event] <- dweibull(
y_oscale[idx_event],
shape = refm_shape_agg,
scale = ilpreds[, idx_event] / gamma(1 + 1 / as.vector(refm_shape_agg)),
log = TRUE
)
}
return(wobs_mat * ll_unw)
}, cens_var = ~ is_censored)
latent_ppd_oscale_weib <- function(
ilpreds_resamp,
dis_resamp = rep(NA, nrow(ilpreds_resamp)),
wobs = rep(1, ncol(ilpreds_resamp)),
cl_ref,
wdraws_ref = rep(1, length(cl_ref)),
idxs_prjdraws
) {
warning("The draws from this `latent_ppd_oscale` function are uncensored.")
refm_shape_agg <- cl_agg(refm_shape, cl = cl_ref, wdraws = wdraws_ref)
refm_shape_agg_resamp <- refm_shape_agg[idxs_prjdraws, , drop = FALSE]
ppd <- rweibull(
prod(dim(ilpreds_resamp)),
shape = refm_shape_agg_resamp,
scale = ilpreds_resamp / gamma(1 + 1 / as.vector(refm_shape_agg_resamp))
)
ppd <- matrix(ppd, nrow = nrow(ilpreds_resamp), ncol = ncol(ilpreds_resamp))
return(ppd)
}
refm_weib <- get_refmodel(
refm_fit_weib,
latent = TRUE,
latent_ll_oscale = latent_ll_oscale_weib,
latent_ppd_oscale = latent_ppd_oscale_weib
)The functions above use the shape draws of the full-data fit
(refm_shape). This is correct for varsel() and
for the LOO CV that we use below. For K-fold CV, see section “K-fold CV with custom response-scale
functions”.
Run cv_varsel():
cvvs_weib <- cv_varsel(
refm_weib,
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0
)Cutting off the search at size 19. Use argument `nterms_max` to customize this.
Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (2 / 500) Pareto k's for the reference model's PSIS-LOO weights
are > 0.7.
Warning: In the recalculation of the latent response values, some (3 / 500) expectation-specific Pareto k-values are > 0.7.
In general, we recommend K-fold CV in this case.
As in the Poisson example, warnings about high Pareto \(\hat{k}\) values can come from the short
MCMC run of this vignette. In a real analysis, we would run more MCMC
iterations, check the Pareto \(\hat{k}\) values again, and use \(K\)-fold CV
(cv_method = "kfold") if many of them are high.
Plot the results:
plot(cvvs_weib, stats = "mlpd", deltas = TRUE)
Hence, the truly relevant predictors are identified correctly.
Based on this plot, we decide for a submodel size of 10:
size_decided_weib <- 10This is not the size that suggest_size() would suggest,
but as mentioned in the main vignette and in the documentation,
suggest_size() provides only a quite heuristic decision (so
we stick with our manual decision here):
suggest_size(cvvs_weib, stat = "mlpd")[1] 9
Project onto the submodel of this size and perform a “posterior-projection predictive check” (PPPC):
predictors_final_weib <- head(ranking(cvvs_weib)[["fulldata"]],
size_decided_weib)
prj_weib <- project(refm_weib, predictor_terms = predictors_final_weib)
prj_predict_weib <- proj_predict(prj_weib)Warning in proj$refmodel$family$latent_ppd_oscale(mu_oscale_resamp, dis_resamp
= proj$dis[draw_inds], : The draws from this `latent_ppd_oscale` function are
uncensored.
bayesplot::ppc_km_overlay(y = dat_sim_weib$yobs, yrep = prj_predict_weib,
status_y = 1 - dat_sim_weib$is_censored)Note: `extrapolation_factor` now defaults to 1.2 (20%).
To display all posterior predictive draws, set `extrapolation_factor = Inf`.

As expected, this PPPC indicates that this submodel is a reasonable one (keeping in mind that such a PPPC is just a single model-diagnostic tool).
Example: Log-normal distribution with right-censored observations
For the log-normal model, we complete our data generation as follows:
sdlog_lnorm <- 0.3
y_lnorm <- rlnorm(N_surv, meanlog = linpreds_surv, sdlog = sdlog_lnorm)
is_event_lnorm <- y_lnorm < cens_surv
yobs_lnorm <- y_lnorm
yobs_lnorm[!is_event_lnorm] <- cens_surv[!is_event_lnorm]
dat_sim_lnorm <- data.frame(yobs = yobs_lnorm,
is_censored = 1 - is_event_lnorm,
dat_sim_surv)Again, we fit a reference model in brms:
refm_fit_lnorm <- brms::brm(
formula = yobs | cens(is_censored) ~ .,
family = brms::lognormal(),
data = dat_sim_lnorm,
prior = brms::prior(R2D2(mean_R2 = 1/3, prec_R2 = 3)),
### Only for the sake of speed (not recommended in general):
chains = 2,
###
silent = 2,
refresh = 0
)The following code prepares the cv_varsel() run and the
downstream projpred steps:
latent_ll_oscale_lnorm <- structure(function(
ilpreds,
dis = rep(NA, nrow(ilpreds)),
y_oscale,
wobs = rep(1, ncol(ilpreds)),
cens,
cl_ref,
wdraws_ref = rep(1, length(cl_ref))
) {
idxs_cens <- which(cens == 1)
idxs_event <- setdiff(seq_along(cens), idxs_cens)
wobs_mat <- matrix(wobs, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
ll_unw <- matrix(nrow = nrow(ilpreds), ncol = ncol(ilpreds))
for (idx_cens in idxs_cens) {
ll_unw[, idx_cens] <- plnorm(
y_oscale[idx_cens],
meanlog = ilpreds[, idx_cens],
sdlog = dis,
lower.tail = FALSE,
log.p = TRUE
)
}
for (idx_event in idxs_event) {
ll_unw[, idx_event] <- dlnorm(
y_oscale[idx_event],
meanlog = ilpreds[, idx_event],
sdlog = dis,
log = TRUE
)
}
return(wobs_mat * ll_unw)
}, cens_var = ~ is_censored)
latent_ppd_oscale_lnorm <- function(
ilpreds_resamp,
dis_resamp = rep(NA, nrow(ilpreds_resamp)),
wobs = rep(1, ncol(ilpreds_resamp)),
cl_ref,
wdraws_ref = rep(1, length(cl_ref)),
idxs_prjdraws
) {
warning("The draws from this `latent_ppd_oscale` function are uncensored.")
ppd <- rlnorm(
prod(dim(ilpreds_resamp)),
meanlog = ilpreds_resamp,
sdlog = dis_resamp
)
ppd <- matrix(ppd, nrow = nrow(ilpreds_resamp), ncol = ncol(ilpreds_resamp))
return(ppd)
}
refm_lnorm <- get_refmodel(
refm_fit_lnorm,
latent = TRUE,
latent_ilink = function(lpreds, cl_ref,
wdraws_ref = rep(1, length(cl_ref))) lpreds,
latent_ll_oscale = latent_ll_oscale_lnorm,
latent_ppd_oscale = latent_ppd_oscale_lnorm,
dis = as.matrix(refm_fit_lnorm)[, "sigma", drop = FALSE]
)Argument latent_ilink returns the latent predictors
unchanged, because the two custom functions use them as
meanlog. Argument dis contains the
sigma draws of the full-data fit. For K-fold CV, see
section “K-fold CV with custom response-scale
functions”.
Run cv_varsel():
cvvs_lnorm <- cv_varsel(
refm_lnorm,
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0
)Cutting off the search at size 19. Use argument `nterms_max` to customize this.
Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (3 / 500) Pareto k's for the reference model's PSIS-LOO weights
are > 0.7.
Warning: In the recalculation of the latent response values, some (8 / 500) expectation-specific Pareto k-values are > 0.7.
In general, we recommend K-fold CV in this case.
As in the Poisson example, warnings about high Pareto \(\hat{k}\) values can come from the short
MCMC run of this vignette. In a real analysis, we would run more MCMC
iterations, check the Pareto \(\hat{k}\) values again, and use \(K\)-fold CV
(cv_method = "kfold") if many of them are high.
Plot the results:
plot(cvvs_lnorm, stats = "mlpd", deltas = TRUE)
Hence, the truly relevant predictors are identified correctly.
Based on this plot, we decide for a submodel size of 10:
size_decided_lnorm <- 10This is also the size that suggest_size() would
suggest:
suggest_size(cvvs_lnorm, stat = "mlpd")[1] 10
Project onto the submodel of this size and perform a “posterior-projection predictive check” (PPPC):
predictors_final_lnorm <- head(ranking(cvvs_lnorm)[["fulldata"]],
size_decided_lnorm)
prj_lnorm <- project(refm_lnorm, predictor_terms = predictors_final_lnorm)
prj_predict_lnorm <- proj_predict(prj_lnorm)Warning in proj$refmodel$family$latent_ppd_oscale(mu_oscale_resamp, dis_resamp
= proj$dis[draw_inds], : The draws from this `latent_ppd_oscale` function are
uncensored.
bayesplot::ppc_km_overlay(y = dat_sim_lnorm$yobs, yrep = prj_predict_lnorm,
status_y = 1 - dat_sim_lnorm$is_censored)Note: `extrapolation_factor` now defaults to 1.2 (20%).
To display all posterior predictive draws, set `extrapolation_factor = Inf`.

As expected, this PPPC indicates that this submodel is a reasonable one (keeping in mind that such a PPPC is just a single model-diagnostic tool).
K-fold CV with custom response-scale functions
In the negative binomial and Weibull examples above, the custom
response-scale functions use posterior draws from the full-data
reference model fit (refm_prec, refm_shape).
In the log-normal example, argument dis contains the
sigma draws of the full-data fit. This is correct for
varsel() and for cv_varsel() with
cv_method = "LOO", because these use only the full-data
fit. With cv_method = "kfold", the reference model is
refitted in each fold, and projpred creates the
fold-wise reference model objects with the function in element
cvrefbuilder of the reference model object. The default
cvrefbuilder functions of
get_refmodel.stanreg() and
brms::get_refmodel.brmsfit() pass the same custom functions
and the same dis to the fold-wise reference model objects,
so the folds use the draws from the full-data fit, meaning that
information from the held-out observations can enter the fold-wise
predictive evaluation.
To use the draws from the fold-wise refits, define the custom
functions inside a function that takes the fitted model object as its
argument and calls get_refmodel(). Then define a function
that calls this outer function with the fold-wise refit. For an
rstanarm reference model, pass this function as
argument cvrefbuilder of get_refmodel(). In
the call inside cvrefbuilder, argument
called_from_cvrefbuilder = TRUE suppresses some warnings
that are already shown for the full-data reference model object (see
?init_refmodel). Method
brms::get_refmodel.brmsfit() does not accept a
cvrefbuilder argument. For a brms
reference model, create the reference model object first, then set its
element cvrefbuilder to this function. The default
cvrefbuilder of brms::get_refmodel.brmsfit()
also passes its arguments resp and brms_seed.
If you use these arguments, pass them in the custom
cvrefbuilder too. For the negative binomial example, this
can look as follows:
get_refmodel_nebin <- function(fit, ...) {
refm_prec <- as.matrix(fit)[, "reciprocal_dispersion", drop = FALSE]
latent_ll_oscale_nebin <- function(ilpreds,
dis = rep(NA, nrow(ilpreds)),
y_oscale,
wobs = rep(1, ncol(ilpreds)),
cens,
cl_ref,
wdraws_ref = rep(1, length(cl_ref))) {
y_oscale_mat <- matrix(y_oscale, nrow = nrow(ilpreds),
ncol = ncol(ilpreds), byrow = TRUE)
wobs_mat <- matrix(wobs, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
refm_prec_agg <- cl_agg(refm_prec, cl = cl_ref, wdraws = wdraws_ref)
ll_unw <- dnbinom(y_oscale_mat, size = refm_prec_agg, mu = ilpreds,
log = TRUE)
return(wobs_mat * ll_unw)
}
latent_ppd_oscale_nebin <- function(
ilpreds_resamp,
dis_resamp = rep(NA, nrow(ilpreds_resamp)),
wobs = rep(1, ncol(ilpreds_resamp)),
cl_ref,
wdraws_ref = rep(1, length(cl_ref)),
idxs_prjdraws
) {
refm_prec_agg <- cl_agg(refm_prec, cl = cl_ref, wdraws = wdraws_ref)
refm_prec_agg_resamp <- refm_prec_agg[idxs_prjdraws, , drop = FALSE]
ppd <- rnbinom(prod(dim(ilpreds_resamp)), size = refm_prec_agg_resamp,
mu = ilpreds_resamp)
ppd <- matrix(ppd, nrow = nrow(ilpreds_resamp), ncol = ncol(ilpreds_resamp))
return(ppd)
}
get_refmodel(fit, latent = TRUE,
latent_ll_oscale = latent_ll_oscale_nebin,
latent_ppd_oscale = latent_ppd_oscale_nebin, ...)
}
refm_nebin_kfold <- get_refmodel_nebin(
refm_fit_nebin,
cvrefbuilder = function(cvfit) {
get_refmodel_nebin(cvfit, called_from_cvrefbuilder = TRUE)
}
)Then cv_varsel(refm_nebin_kfold, cv_method = "kfold")
uses the reciprocal_dispersion draws of the fold-wise refit
in each fold. We do not run K-fold CV here, because it refits the
reference model \(K\) times. For the
Weibull example, define refm_shape inside such a function
in the same way. For the log-normal example, pass
dis = as.matrix(fit)[, "sigma", drop = FALSE] to
get_refmodel() inside such a function.