Need help with modelselection?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

avehtari
137 Stars 31 Forks 92 Commits 1 Opened issues

Description

Tutorial on model assessment, model selection and inference after model selection

Services available

!
?

Need anything else?

Contributors list

No Data

Model assesment, selection and inference after selection

Example notebooks in R using rstanarm, rstan, bayesplot, loo, projpred.

Talks

Outline of the StanCon 2018 Asilomar tutorial and links to notebooks

  • Basics of predictive performance estimation
  • When cross-validation is not needed
  • When cross-validation is useful
    • We don't trust the model - roaches
    • Complex model with posterior dependencies - collinear
  • On accuracy of cross-validation
  • Cross-validation and hierarchical models
  • When cross-validation is not enough
  • loo 2.0
  • Projection predictive model selection

Additional case studies

See also

References

  • Afrabandpey, H., Peltola, T., Piironen, J., Vehtari, A., and Kaski, S. (2019). Making Bayesian predictive models interpretable: A decision theoretic approach. arXiv preprint arXiv:1910.09358
  • Bürkner, P.-C., Gabry, J., Vehtari, A. (2018). Leave-one-out cross-validation for non-factorizable normal models. arXiv:1810.10559
  • Bürkner, P.-C., Gabry, J., Vehtari, A. (2020). Approximate leave-future-out cross-validation for time series models. Journal of Statistical Computation and Simulation, doi:10.1080/00949655.2020.1783262. Online. Preprint arXiv:1902.06281
  • Gelman, A., Hwang, J., and Vehtari, A. (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6):997–1016. Preprint
  • Gelman, A., Goodrich, B., Gabry, J., and Vehtari, A. (2018). R-squared for Bayesian regression models. The American Statistician, doi:10.1080/00031305.2018.1549100. Online.
  • Magnusson, M., Andersen, M.R., Jonasson, J., Vehtari, A. (2019). Bayesian leave-one-out cross-validation for large data. Thirty-sixth International Conference on Machine Learning, PMLR 97:4244--4253. Online.
  • * Magnusson, M., Andersen, M.R., Jonasson, J., Vehtari, A. (2020). Leave-one-out cross-validation for Bayesian model comparison in large data. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 108:341-351. Online. preprint arXiv:2001.00980.
  • Paananen, T., Piironen, J., Bürkner, P.-C., and Vehtari, A. (2020). Implicitly adaptive importance sampling. arXiv:1906.08850
  • Pavone, F., Piironen, J., Bürkner, P.-C., and Vehtari, A- (2020). Using reference models in variable selection. arXiv preprint arXiv:2004.13118
  • Piironen, J. and Vehtari, A. (2016), Comparison of Bayesian predictive methods for model selection, Statistics and Computing 27(3), 711–735. Online
  • Piironen, J., and Vehtari, A. (2017). On the hyperprior choice for the global shrinkage parameter in the horseshoe prior. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:905-913. Online
  • Piironen, J., and Vehtari, A. (2017). Sparsity information and regularization in the horseshoe and other shrinkage priors. In Electronic Journal of Statistics, 11(2):5018-5051. Online
  • Piironen, J., and Vehtari, A. (2018). Iterative supervised principal components. Proceedings of the 21th International Conference on Artificial Intelligence and Statistics, accepted for publication. arXiv preprint arXiv:1710.06229
  • Piironen, J., Paasiniemi, M., and Vehtari, A. (2020). Projective Inference in High-dimensional Problems: Prediction and Feature Selection. Electronic Journal of Statistics, 14(1):2155-2197. Online. Preprint arXiv:1810.02406
  • Vehtari, A., Gelman, A., Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5):1413–1432. arXiv preprint.
  • Vehtari, A., Simpson, D., Gelman, A., Yao, Y., and Gabry, J. (2019). Pareto smoothed importance sampling. arXiv preprint.
  • Vehtari, A., Mononen, T., Tolvanen, V., and Winther, O. (2016). Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models. JMLR, 17(103):1–38. Online
  • Vehtari, A. and Ojanen, J.: 2012, A survey of Bayesian predictive methods for model assessment, selection and comparison, Statistics Surveys 6, 142–228. Online
  • Williams, D. R., Piironen, J., Vehtari, A., and Rast, P. (2018). Bayesian estimation of Gaussian graphical models with projection predictive selection. arXiv:1801.05725
  • Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2017). Using stacking to average Bayesian predictive distributions. In Bayesian Analysis, doi:10.1214/17-BA1091, Online

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.