track three of eight

RLHF

About 18 hours, 17% of the whole. 5 lectures, 2 with an exercise.

Figure.The reward model was trained on people picking A over B.

People compare two outputs, a reward model learns from the comparisons, and PPO, DPO, and their relatives optimize against it.

Start with lecture 09. Each lecture lists what to read first, and ends with a checkpoint: something to do with the page closed.

09Reward modeling for RLHF2–3 h
checkpointExplain how a preference dataset becomes a scalar reward model (Bradley-Terry), and how reward models get gamed.
exercise← start here
10PPO for language models (the full RLHF pipeline)3–4 h
checkpointWalk through the RLHF loop end to end: SFT → reward model → PPO with a KL penalty to the reference policy.
video
11Direct Preference Optimization (DPO)2–3 h
checkpointExplain how DPO skips the explicit reward model and what it's implicitly optimizing.
exercise
12Beyond DPO (GRPO, RRHF, IPO, and post-DPO methods)2–4 h
checkpointSay what GRPO changes relative to PPO and why that matters for reasoning-style training.
17Online and Iterative Preference Optimization2–3 h
checkpointExplain why offline DPO underperforms PPO-RLHF, and what iterating the loop fixes.

Exercises in this track

09-reward-model: Bradley-Terry reward model on synthetic preferences (with lecture 09)
11-dpo: DPO on a toy preference dataset (with lecture 11)