Classical RL

The environment gives the reward. Off-policy and model-based methods, exploration, multi-agent RL, world models, meta-RL, robotics.

Start with lecture 07. Each lecture lists what to read first, and ends with a checkpoint: something to do with the page closed.

1. 07Off-Policy Learning - SAC and TD32–3 h

   checkpointExplain what a replay buffer buys you and why off-policy methods need extra machinery to stay stable.

   ← start here

2. 08Model-based reinforcement learning2–3 h

   checkpointExplain when learning a model is worth it and how model error bites you.

   video

3. 20Exploration: from ε-greedy to intrinsic motivation~3 h + exercise

   checkpointExplain why uniform random exploration fails on Montezuma's Revenge and what RND's prediction-error bonus computes. Then do the exercise: exercises/20-exploration/ (RND on a sparse-reward chain MDP, tests green).

   exercisevideo

4. 21Multi-agent RL and self-play~3 h

   checkpointExplain why naive independent Q-learning fails when other agents learn too, and what fictitious self-play buys you in non-transitive games.

   video

5. 22World models~3 h

   checkpointExplain the latent-dynamics premise (Dreamer family) and how MuZero learns a model that's only as accurate as needed for planning.

   video

6. 32Meta-RL and in-context RL2–3 h

   checkpointExplain how MAML differs from RL² in *where* the adaptation happens, and what Algorithm Distillation learns from the histories of an RL algorithm.

7. 33Robotics RL2–3 h

   checkpointExplain why off-policy methods dominate real-robot work and what a VLA model contributes vs. a from-scratch RL policy.

Exercises in this track

• 20-exploration: RND on a sparse-reward chain MDP (with lecture 20)

On video

David Silver, Lecture 8: Integrating Learning and Planning (with 08)CS 285, Lecture 12: Model-Based RL with Policies (with 08)David Silver, Lecture 9: Exploration and Exploitation (with 20)David Silver, Lecture 10: Classic Games (with 21)