General Prerequisites: None, but some basic knowledge of stochastic calculus (as would be obtained from B8.2 Continuous Martingales and Stochastic Calculus or B8.3 Mathematical Models of Financial Derivatives) will be assumed. Familiarity with applied PDE (as would be obtained from B5.2 Applied Partial Differential Equations, B6.1 Numerical Solution of Partial Differential Equations, or B7.1 Classical Mechanics) would also be beneficial.
Course Overview: Optimal control is the question of how one should select actions sequentially through time. The problem appears, in various forms, in many applications, from industrial problems and classical mechanics through to problems in biology and finance. This course will study the mathematics required to understand these problems, both in discrete and continuous time, and in settings with and without randomness. The two main perspectives on control -- dynamic programming and the Pontryagin principle -- will be explored, along with how these perspectives lead to equations that describe the optimal action. The numerical solution of these equations will also be considered.
Learning Outcomes: The students will develop an understanding of the classical theory of optimal control, and be able to determine optimal controls within mathematical models. They will be familiar with manipulating the corresponding PDEs, and with reinforcement learning techniques to solve them.
Course Synopsis: Dynamic programming in discrete time, the Bellman equation and value function. Iteration methods for discrete systems and variations from reinforcement learning. Continuous deterministic systems and the Hamilton--Jacobi equation. Pontryagin maximum principle for deterministic systems. Feynman--Kac theorem and values in stochastic systems, the Linear-Quadratic-Gaussian case. Martingale characterization of optimality and the Hamilton--Jacobi--Bellman equation, maximum principle and verification theorem. Examples from finance and engineering.
