Bayesian Statistical Learning

Lecturers

Dr Matias Quiroz, University of Technology Sydney
Dr Aishwarya Bhaskaran, University of New South Wales

Synopsis

This subject introduces the Bayesian approach to statistical inference, which has become widely used in Data Science and Machine Learning. The aim is to provide a solid introduction to Bayesian data analysis through a combination of theoretical and methodological concepts, with a strong emphasis on the computer implementation of modern simulation algorithms.

The subject is organised into four modules. The first module introduces the Bayesian paradigm and develops inferential tools for simple statistical models. The second module considers more advanced models, including linear regression, spline regression, and classification models. The third module focuses on computation and presents modern algorithms used to carry out Bayesian inference. Finally, the fourth module covers model comparison techniques and advanced topics such as Bayesian variable selection and hierarchical models.

In each module, students are introduced to the underlying theory and methodology, followed by a workshop in which practical problems are solved using a computer. Students gain experience with modern Bayesian methods that can be applied to complex problems in Data Science and Machine Learning. The subject places strong emphasis on the practical implementation of the methods presented. To strengthen conceptual understanding, students will implement several of the core algorithms themselves rather than relying solely on existing software packages. The recommended programming language for the workshops is R, although students are free to use any software of their choice (e.g. Python or Julia).

Course Overview

Each lecture is 2 hours (20 total) and each workshop is 2 hours (8 total). Total 28 hours.

Week 1: Foundations

  • Lecture 1: The Bayesian paradigm. Single parameter models. Conjugate priors. Prior elicitation. Noninformative priors. Jeffreys’ prior.
  • Lecture 2: Bayesian prediction. Bayesian inference as a decision theory problem. Multiparameter models. Analytic marginalisation. Marginalisation via simulation.
  • Workshop 1.

Week 2: Regression models

  • Lecture 3: Bayesian linear regression. Shrinkage priors. Bayesian spline regression.
  • Lecture 4: Bayesian classification. Asymptotics. Normal approximation. Generative models (naïve Bayes). Discriminative models (logistic regression).
  • Lecture 5: Bayesian prediction. Connections to decision theory.
  • Workshop 2.

Week 3: Bayesian computation

  • Lecture 6: Monte Carlo integration. Importance sampling. Inverse CDF method. Rejection sampling.
  • Lecture 7: Gibbs sampling and data augmentation. Markov processes. The Gibbs sampler. Data augmentation.
  • Lecture 8: The Metropolis and Metropolis–Hastings sampler. Efficiency of simulation. Assessing convergence.
  • Workshop 3.

Week 4: Model inference and hierarchical models

  • Lecture 9: Marginal likelihoods. Bayesian model averaging. Bayesian variable selection. Posterior predictive model evaluation.
  • Lecture 10: Hierarchical models. Pooling estimates. MCMC sampling with RStan.
  • Workshop 4.

Prerequisites & Pre-reading 

This subject is suitable for students who have taken at least one course in statistics above the introductory level and have some programming experience. The students should be confident in their knowledge of concepts such as integrals, derivatives, probability distributions, conditional probability and expectations. The students should have some basic knowledge in linear algebra, including matrix operations such as multiplication and inverses, and concepts such as eigenvalues and eigenvectors. A good and compact text of the mathematical tools used in this subject can be found here: However, note that we will only encounter a subset of these, so students are not expected to know the whole content.

This subject is intended as a foundational course in Bayesian statistical learning for students at the Honours or Masters level. It introduces the core principles of Bayesian inference and modern computational methods used in data science and machine learning. The subject assumes prior knowledge of probability, mathematical statistics, and regression modelling.

Assessment

  • TBC

Not sure if you should sign up for this course?

Take this pre-enrolment QUIZ to self evaluate and get a measure of the key foundational knowledge required.

QUIZ SOLUTIONS 

matias_headshot

Dr Matias Quiroz, University of Technology, Sydney

Matias Quiroz is a Senior Lecturer in Statistics in the School of Mathematical and Physical Sciences at the University of Technology Sydney. His research focuses on scalable Bayesian computation, Monte Carlo methods, and variational inference. He develops computational methods that make modern Bayesian inference feasible for large and complex datasets. Passionate about all things Bayesian, he regularly preaches the virtues of the Bayesian paradigm to his students. Matias is an experienced educator committed to making advanced statistical ideas accessible and engaging. You can find out more about him at www.matiasquiroz.com.

aishwarya_headshot

Dr Aishwarya Bhaskaran, University of New South Wales

Aishwarya Bhaskaran is a Lecturer in Statistics in the School of Mathematics and Statistics at the University of New South Wales. She received her PhD in Statistics from the University of Technology Sydney and subsequently completed a postdoctoral fellowship in Statistics at Macquarie University. Her research primarily focuses on likelihood-based inference, with particular emphasis on generalised linear mixed models, semi-parametric methods for survival analysis, penalised methods and asymptotic theory. She has also worked on Bayesian inference using variational approximation methods, with a particular focus on improving variational inference for complex models and applying these methods to large datasets.