Everything in the study repo, on one page. Lectures are grouped by track, in the order the curriculum suggests.
Tracks
- Foundations: MDPs, value functions, policy gradients, actor-critic, trust regions, PPO. Every other track builds on these.
- Classical RL: The environment gives the reward. Off-policy and model-based methods, exploration, multi-agent RL, world models, meta-RL, robotics.
- RLHF: People compare two outputs, a reward model learns from the comparisons, and PPO, DPO, and their relatives optimize against it.
- RLAIF: An LLM judge or a written constitution stands in for the human labeler.
- RLVR and reasoning: A checker decides: the answer matches, the tests pass. Reasoning models, process vs outcome rewards, and how verifiers get gamed.
- Agentic RL: Multi-step tasks with tools, browsers, and code. The reward usually arrives at the end, so credit assignment gets hard.
- Offline RL: Learning from a fixed dataset with no new interaction, including imitation and distillation.
- Systems: How LLM-RL training runs: distributed rollouts and learners, inference engines, hardware.
Lectures
Foundations
- Lecture 01: MDPs and Bellman Equations, 2–3 h · exercise
- Lecture 02: Policy Gradients from Scratch, 3–4 h · exercise
- Lecture 03: Value Functions & Q-Learning, 2–3 h · exercise
- Lecture 04: Actor-Critic Methods, 2–3 h · exercise
- Lecture 05: Trust Region Policy Optimization (TRPO), 2–4 h
- Lecture 06: PPO - Proximal Policy Optimization, 2–3 h · exercise
Classical RL
- Lecture 07: Off-Policy Learning - SAC and TD3, 2–3 h
- Lecture 08: Model-based reinforcement learning, 2–3 h
- Lecture 20: Exploration: from ε-greedy to intrinsic motivation, ~3 h · exercise
- Lecture 21: Multi-agent RL and self-play, ~3 h
- Lecture 22: World models, ~3 h
- Lecture 32: Meta-RL and in-context RL, 2–3 h
- Lecture 33: Robotics RL, 2–3 h
RLHF
- Lecture 09: Reward modeling for RLHF, 2–3 h · exercise
- Lecture 10: PPO for language models (the full RLHF pipeline), 3–4 h
- Lecture 11: Direct Preference Optimization (DPO), 2–3 h · exercise
- Lecture 12: Beyond DPO (GRPO, RRHF, IPO, and post-DPO methods), 2–4 h
- Lecture 17: Online and Iterative Preference Optimization, 2–3 h
RLAIF
- Lecture 14: Constitutional AI, RLAIF, and Self-Improvement, 2–3 h
- Lecture 27: RLAIF and synthetic preferences at scale, 2–3 h
- Lecture 34: Self-distillation and self-improvement loops, 2–3 h
RLVR and reasoning
- Lecture 13: RLHF for Code Generation, 2–3 h
- Lecture 15: RL with Verifiable Rewards and Reasoning Models, 3–4 h · exercise
- Lecture 23: Process reward models vs outcome reward models, 2–3 h
- Lecture 26: RL for mathematical reasoning, ~3 h
- Lecture 28: Reward hacking and verifier design, 2–3 h
Agentic RL
- Lecture 16: Agentic RL: tool use and multi-turn interaction, ~3 h
- Lecture 24: Computer use and browser agents, 2–3 h
- Lecture 25: Long-horizon credit assignment, ~3 h
Offline RL
Systems
- Lecture 29: Distributed RL systems, 2–3 h
- Lecture 30: RL inference infrastructure for LLMs, ~2 h
- Lecture 31: Hardware for RL, ~2 h
Exercises
- 01-mdps: value iteration on a gridworld (with lecture 01)
- 02-policy-gradients: REINFORCE, solve CartPole (with lecture 02)
- 03-q-learning: tabular Q-learning on FrozenLake (with lecture 03)
- 04-actor-critic: actor-critic with a learned baseline, on CartPole (with lecture 04)
- 05-ppo: PPO with GAE, solve CartPole (with lecture 06)
- 09-reward-model: Bradley-Terry reward model on synthetic preferences (with lecture 09)
- 11-dpo: DPO on a toy preference dataset (with lecture 11)
- 15-grpo-rlvr: GRPO on a verifiable arithmetic toy task (with lecture 15)
- 20-exploration: RND on a sparse-reward chain MDP (with lecture 20)
Homework
- How the homework works
- HW1: MDPs, Bellman equations, value iteration: problems, solutions
- hw02 policy gradients (planned)
- hw03 q learning and dqn (planned)
- hw04 actor critic and ppo (planned)
- hw05 rlhf and dpo (planned)
- hw06 grpo and rlvr (planned)
- hw07 agentic or offline (planned)
Cheat sheets
- KL control in RL: cheat sheet
- RL-for-LLM training loops: 2024/2025 map
- Deep RL Mathematical Formulas - Quick Reference
- Deep RL Quick Reference & Cheat Sheet
- RL loss functions: cheat sheet
- RLHF vs DPO vs GRPO vs RLVR: cheat sheet
Diagrams
Reading lists
- Overview
- Reference: agentic RL: tool use, computer use, multi-turn
- Reference: GRPO and RL with verifiable rewards
- RL-enhanced LLM code generation · all collected papers
- RL for program synthesis with LLMs · all collected papers
- RLHF and alignment · all collected papers