Offline RL

Learning from a fixed dataset with no new interaction, including imitation and distillation.

Start with lecture 18. Each lecture lists what to read first, and ends with a checkpoint: something to do with the page closed.

1. 18Distillation of Reasoning Models~2 h

   checkpointExplain the R1-distill recipe and why a small model can imitate a big reasoner's chains but couldn't discover them via RL alone.

   video← start here

2. 19Offline RL2–3 h

   checkpointExplain why naive Q-learning on a fixed dataset produces optimistic Q-values for out-of-distribution actions, and how IQL avoids the problem.

   video

On video

CS 285, Lecture 2: Imitation Learning (with 18)CS 285, Lecture 15: Offline Reinforcement Learning (with 19)CS 285, Lecture 16: Offline Reinforcement Learning 2 (with 19)