RLVR and reasoning

A checker decides: the answer matches, the tests pass. Reasoning models, process vs outcome rewards, and how verifiers get gamed.

Start with lecture 13. Each lecture lists what to read first, and ends with a checkpoint: something to do with the page closed.

1. 13RLHF for Code Generation2–3 h

   checkpointExplain why unit-test pass/fail is a good reward signal for code and where it falls short.

   ← start here

2. 15RL with Verifiable Rewards and Reasoning Models3–4 h

   checkpointWrite GRPO's group-relative advantage and say why it removes the need for a value network; sketch the R1-Zero loop. Then do the exercise: exercises/15-grpo-rlvr/ (implement GRPO on a verifiable toy task).

   exercise

3. 23Process reward models vs outcome reward models2–3 h

   checkpointExplain why PRMs reduce noisy credit assignment but add a gameable surface, and why DeepSeek chose to skip them.

4. 26RL for mathematical reasoning~3 h

   checkpointWalk through the DeepSeek-R1 training recipe: dataset, reward design, GRPO loop; explain the "aha moment" finding.

5. 28Reward hacking and verifier design2–3 h

   checkpointDescribe one concrete reward-hack in code-RL and one in math-RL; explain why holding out a verifier set is analogous to holding out test data.

Exercises in this track

• 15-grpo-rlvr: GRPO on a verifiable arithmetic toy task (with lecture 15)