Skip to contents

Add relative model fit criteria to model objects

Source: R/add-criterion.R
add_criterion.Rd

Add leave-one-out (LOO; Vehtari et al., 2017) and/or widely applicable information criteria (WAIC; Watanabe, 2010) to fitted model objects.

Usage

add_criterion(
  x,
  criterion = c("loo", "waic"),
  overwrite = FALSE,
  save = TRUE,
  ...,
  r_eff = NA
)

Arguments

x

A measrfit object.

criterion

A vector of criteria to calculate and add to the model object.

overwrite

Logical. Indicates whether existing criteria should be overwritten. For example, if LOO has already been added to the model object add_criterion() is called again with criterion = "loo", should LOO be recalculated? Default is FALSE.

save

Logical. Only relevant if a file was specified in the measrfit object passed to x. If TRUE (the default), the model is re-saved to the specified file when new criteria are added to the R object. If FALSE, the new criteria will be added to the R object, but the saved file will not be updated.

...

Additional arguments passed to loo::loo.array() or loo::waic.array().

r_eff

Vector of relative effective sample size estimates for the likelihood (exp(log_lik)) of each observation. This is related to the relative efficiency of estimating the normalizing term in self-normalizing importance sampling when using posterior draws obtained with MCMC. If MCMC draws are used and r_eff is not provided then the reported PSIS effective sample sizes and Monte Carlo error estimates will be over-optimistic. If the posterior draws are independent then r_eff=1 and can be omitted. The warning message thrown when r_eff is not specified can be disabled by setting r_eff to NA. See the relative_eff() helper functions for computing r_eff.

Value

A modified measrfit object with the criteria slot populated with the specified criteria.

References

Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413-1432. https://doi.org/10.1007/s11222-016-9696-4

Watanabe, S. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11(116), 3571-3594. http://jmlr.org/papers/v11/watanabe10a.html