Stochastic Control, Reinforcement Learning and Applications

Lecturers

Dr Libo Li, UNSW Sydney
Dr Ruyi Liu, UNSW Sydney

Synopsis

This course provides a unified mathematical framework for decision-making under uncertainty, bridging the gap between classical Optimal Control and modern Reinforcement Learning (RL). Over four weeks, students progress from the foundations of Dynamic Programming and the Linear Quadratic Regulator (LQR) to Stochastic Differential Equations (SDEs) and the Stochastic Hamilton-Jacobi-Bellman (HJB) equation. The course will then go into Markov Decision Processes (MDPs) to establish the logic of discrete-state stochasticity, culminating in model-free Reinforcement Learning (Q-learning and Policy Function Approximation).

Course Overview

Week 1: Deterministic (Discrete and Continuous Time) Optimal Control

  • The Principle of Optimality and Dynamic Programming:
  • Bellman Equation: Finite horizons and the backward recursion algorithm.
  • The LQR Problem: Solving the discrete Riccati equation to find the optimal gain “K”.
  • Practical: Coding a 1D LQR controller for a deterministic system.

Week 2: Stochastic Control & The HJB

  • Itô’s Lemma (Simplified): The “stochastic chain rule” needed to differentiate functions of random variables.
  • Controlled Diffusions: Modeling systems using controlled Stochastic Differential Equations.
  • The Stochastic DP and HJB: Deriving the second-order PDE. Understanding how noise σ “smears” the value function.
  • Practical: Comparing a deterministic trajectory vs. a stochastic trajectories.

Week 3: Markov Decision Processes (MDPs)

  • Markov chains and MDP Framework:
  • Policy vs. Value: How to evaluate a fixed behavior in a random environment.
  • Value Iteration: An iterative algorithm to solve for the optimal value function
  • Bellman Operators: Understanding the “contraction mapping” that ensures the algorithm converges.

Week 4: Reinforcement Learning (RL)

  • Relations Between DP and RL
  • Temporal Difference (TD) Learning: Learning from the “error” between successive estimates.
  • Q-Learning: Removing the need for a transition model by learning “Quality” values for action-state pairs.
  • Exploration (Epsilon-Greedy): Why the agent must occasionally act sub-optimally to discover better paths.
  • Policy Function Approximation: Using Neural Networks to represent the Q-Table for high-dimensional states.

Prerequisites 

  • Calculus (level 1/2)
  • Linear Algebra (level 1/2)
  • Probability (level 2)
  • Basic knowledge of Matlab or Python.

Assessment

  • TBC

Resources/pre-reading

Not sure if you should take this course?

Take this pre-enrolment QUIZ to self evaluate and get a measure of the key foundational knowledge required.

Dr Libo Li

Dr Libo Li, UNSW Sydney

Dr Libo Li is a Senior Lecturer in Statistics in the School of Mathematics and Statistics at the University of New South Wales (UNSW). His research lies at the intersection of probability theory, stochastic analysis, and mathematical finance. His work focuses on theory of stochastic processes, stochastic control, stochastic differential equations, backward stochastic differential equations (BSDEs), optimal stopping, and numerical methods for stochastic systems. More recently, his research has expanded to reinforcement learning and machine learning approaches to stochastic optimisation and mathematical finance.

Dr Ruyi Liu

Dr Ruyi Liu, UNSW Sydney

Dr Ruyi Liu is a Lecturer in the School of Mathematics and Statistics at UNSW Sydney. He works on stochastic control, optimal stopping, and backward stochastic differential equations (BSDEs/FBSDEs), with a particular interest in how these tools apply to real-world markets — from derivatives pricing to trading strategies and electricity systems. A common thread in his work is turning these problems into explicit, implementable solutions, such as threshold trading rules, closed-form optimality, and related BSDEs for pricing and hedging.