unreviewed draftNo one has checked this end to end. Treat the math, code, and citations as unverified. Source on GitHub.
Topic: Reward modeling, RLHF, DPO. Goes with: lecture L09-L11. Status: planned. To be built next.
The plan: derive the Bradley-Terry model from preference data; show how DPO falls out of the optimal RLHF policy with a KL constraint; implement a small DPO loop on a toy preference dataset.