Foundations

MDPs, value functions, policy gradients, actor-critic, trust regions, PPO. Every other track builds on these.

Start with lecture 01. Each lecture lists what to read first, and ends with a checkpoint: something to do with the page closed.

1. 01MDPs and Bellman Equations2–3 h

   checkpointWrite down both Bellman equations and explain what value iteration does, without notes.

   exercisevideo← start here

2. 02Policy Gradients from Scratch3–4 h + exercise

   checkpointDerive ∇J = E[∇ log π · R] and say why the environment dynamics drop out. Then do the exercise: exercises/02-policy-gradients/ (implement REINFORCE, get CartPole solved, tests green).

   exercisevideo

3. 03Value Functions & Q-Learning2–3 h

   checkpointExplain when you'd reach for a value-based method instead of a policy-gradient one.

   exercisevideo

4. 04Actor-Critic Methods2–3 h

   checkpointExplain why subtracting a learned baseline reduces variance without biasing the gradient.

   exercisevideo

5. 05Trust Region Policy Optimization (TRPO)2–4 h

   checkpointExplain what "trust region" means here and what problem it's solving.

6. 06PPO - Proximal Policy Optimization2–3 h

   checkpointWrite the clipped surrogate objective and explain what the clip does and why.

   exercise

Exercises in this track

• 01-mdps: value iteration on a gridworld (with lecture 01)
• 02-policy-gradients: REINFORCE, solve CartPole (with lecture 02)
• 03-q-learning: tabular Q-learning on FrozenLake (with lecture 03)
• 04-actor-critic: actor-critic with a learned baseline, on CartPole (with lecture 04)
• 05-ppo: PPO with GAE, solve CartPole (with lecture 06)

On video

David Silver, Lecture 2: Markov Decision Process (with 01)David Silver, Lecture 3: Planning by Dynamic Programming (with 01)David Silver, Lecture 7: Policy Gradient Methods (with 02)David Silver, Lecture 4: Model-Free Prediction (with 03)David Silver, Lecture 5: Model Free Control (with 03)David Silver, Lecture 6: Value Function Approximation (with 03)