Machine Learning/Deep Learning

Lecturers

Dr Susan Wei, The University of Melbourne
Dr Pavel Krupskiy, The University of Melbourne
Dr Matthew Tam, The University of Melbourne

Synopsis

This course is intended to introduce the foundational theory of machine learning with a special focus on deep learning. Participants will be exposed to a range of tools that will be equally applicable in either academic or industry contexts. The intended learning outcome is that participants will be able to deploy deep learning in an end-to-end fashion and appreciate the theoretical mysteries that deep learning poses to classical statistical learning theory.

Course overview

Week 1: The basics of machine learning using the linear neural networks as a guiding example

• These linear neural networks can be regarded as shallow neural networks and allow us to focus on the basics of neural network training without getting bogged down by complex architectures
• How to choose the loss function, how to train, and how to evaluate the performance
• Linear neural networks for both regression and classification

Week 2: Stochastic optimisation schemes which serve as the workhorse of modern deep learning

• gradient descent, stochastic gradient descent, minibatch stochastic gradient descent
• more advanced stochastic optimisation schemes including momentum, adagrad, RMSProp, Adadelta

Week 3: How to design and train deep neural networks end-to-end on real datasets

• The simplest deep network — the multilayer perception
• How automatic differentiation calculates gradients
• Convolution neural networks which are purpose built for computer vision tasks, one of the crowning achievements of modern deep learning

Week 4: Classical statistical learning theory and its limitations in explaining the unreasonable effectiveness of deep learning.

• Recent advancements in learning theory that look promising for laying a theoretical foundation of deep learning
• Topics covered include loss landscapes of neural networks and inductive biases due to algorithmic regularisation.

Prerequisites

• At a minimum, the successful student will have taken courses in calculus, statistics and linear algebra
• Familiarity with the basics of programming (an introductory course or equivalent experience in C++, Julia, Python, Matlab, or similar should suffice)
• Detailed instructions will be provided in advance of the class for participants to install Python and deep learning libraries such as PyTorch
• Pre-read Chapters 1 and 2 of dl2.ai

Assessment

• 4 weekly quizzes (20%)
• 4 weekly assignments (40%)
• Take home exam (40%)

Attendance requirements

Participation in all lectures and tutorials is expected.

For those completing the subject for their own knowledge/interest, evidence of at least 80% attendance at lectures and tutorials is required to receive a certificate of attendance.

Resources/pre-reading

• Dive Into Deep Learning online textbook, dl2.ai. Pre-read chapters 1 and 2

Not sure if you should sign up for this course?

Take this QUIZ to self-evaluate and get a measure of the key foundational knowledge required.