Notes, lectures, and exercises for learning reinforcement learning and how it is used to train language models.https://rl.yad.codes/enhttps://rl.yad.codes/posts/20-exploration/https://rl.yad.codes/posts/20-exploration/Lecture 01 set up the MDP and the Bellman equations under the assumption that you already had a way to estimate Q(s, a) for every state-action pair you cared about. Lecture 03 introduced Q-learning and gestured at…Tue, 30 Jun 2026 00:00:00 GMTclassicalhttps://rl.yad.codes/posts/21-multi-agent-rl/https://rl.yad.codes/posts/21-multi-agent-rl/Lectures 01–08 treat the world as a single MDP. There's an environment with stationary dynamics, a reward function defined on (state, action), and a single agent whose only job is to act well in it. The textbook (Sutton…Tue, 30 Jun 2026 00:00:00 GMTclassicalhttps://rl.yad.codes/posts/22-world-models/https://rl.yad.codes/posts/22-world-models/Lecture 08: Model-based RL introduced the basic idea: learn a model of environment dynamics, then use it to plan or to generate synthetic training data. That lecture covered Dyna-Q, MBPO with short rollouts from an…Tue, 30 Jun 2026 00:00:00 GMTclassicalhttps://rl.yad.codes/posts/23-process-reward-models/https://rl.yad.codes/posts/23-process-reward-models/Lecture 09 covered learned reward models trained on preference comparisons. Lecture 15 covered verifiable rewards: rule-based checkers that score the final answer of a reasoning chain. Both are outcome signals: the…Tue, 30 Jun 2026 00:00:00 GMTrlvrhttps://rl.yad.codes/posts/24-computer-use-agents/https://rl.yad.codes/posts/24-computer-use-agents/Lecture 16 covered agentic RL through a text-mediated interface: the model emits a string like edit_file("foo.py", ...), an environment executes it, and a textual observation comes back. The action space is finite (a…Tue, 30 Jun 2026 00:00:00 GMTagentichttps://rl.yad.codes/posts/25-long-horizon-credit/https://rl.yad.codes/posts/25-long-horizon-credit/Lecture 16 set up the agentic RL loop: a policy emits an action, an environment returns an observation, the history grows, and a terminal reward shows up at the end. The example episodes there were short (a 3-action…Tue, 30 Jun 2026 00:00:00 GMTagentichttps://rl.yad.codes/posts/26-rl-math-reasoning/https://rl.yad.codes/posts/26-rl-math-reasoning/Lecture 15 covered the general shape of RL with verifiable rewards: a checker replaces a learned reward model, GRPO replaces PPO's critic, and the binary correctness signal turns out to be enough to shape long-horizon…Tue, 30 Jun 2026 00:00:00 GMTrlvrhttps://rl.yad.codes/posts/27-rlaif/https://rl.yad.codes/posts/27-rlaif/Lecture 09 walked through reward modeling with human preference pairs. Lecture 14 introduced RLAIF as one of several AI-feedback techniques and sketched the Constitutional AI pipeline. This lecture stays narrow on a…Tue, 30 Jun 2026 00:00:00 GMTrlaifhttps://rl.yad.codes/posts/28-reward-hacking/https://rl.yad.codes/posts/28-reward-hacking/Every lecture in the LLM block has been an honest attempt to specify what "good" means. Lecture 09 used pairwise human preferences and a Bradley-Terry head. Lecture 14 replaced the human labeler with another model and a…Tue, 30 Jun 2026 00:00:00 GMTrlvrhttps://rl.yad.codes/posts/29-distributed-rl-systems/https://rl.yad.codes/posts/29-distributed-rl-systems/Supervised training has a clean separation: there's a dataset on disk, a loader streams batches to GPUs, and the gradient computation dominates the loop. You can scale by adding more workers reading from the same…Tue, 30 Jun 2026 00:00:00 GMTsystemshttps://rl.yad.codes/posts/30-rl-inference-infra/https://rl.yad.codes/posts/30-rl-inference-infra/Profile a GRPO or PPO run on a modern LLM (say, a 7B-to-70B policy with a group size of 32 and a few thousand prompts per step). You will find that something like 70–90% of wall-clock time goes to generation, running…Tue, 30 Jun 2026 00:00:00 GMTsystemshttps://rl.yad.codes/posts/31-hardware-for-rl/https://rl.yad.codes/posts/31-hardware-for-rl/Almost every other lecture in this series treats compute as a free variable. You write loss.backward() and assume the hardware sorts itself out. That works fine through Lecture 08: CartPole and small MuJoCo tasks run on…Tue, 30 Jun 2026 00:00:00 GMTsystemshttps://rl.yad.codes/posts/32-meta-rl-in-context/https://rl.yad.codes/posts/32-meta-rl-in-context/Every algorithm in lectures 02 through 08 trains on one task. Pick CartPole, pick HalfCheetah, pick Atari Breakout. You put a policy through millions of environment steps, and the thing it ends up doing is "solve this…Tue, 30 Jun 2026 00:00:00 GMTclassicalhttps://rl.yad.codes/posts/33-robotics-rl/https://rl.yad.codes/posts/33-robotics-rl/Everything so far has assumed the agent lives in a simulator or in a text stream. Atari resets in microseconds. A GSM8K problem hands you a clean prompt and a clean answer. PPO runs across thousands of parallel…Tue, 30 Jun 2026 00:00:00 GMTclassicalhttps://rl.yad.codes/posts/34-self-distillation/https://rl.yad.codes/posts/34-self-distillation/Lecture 18: Distillation of Reasoning Models covered a specific recipe: take a large reasoning model, sample reasoning chains, filter by a verifiable checker, train a smaller model on the kept chains. The teacher and…Tue, 30 Jun 2026 00:00:00 GMTrlaifhttps://rl.yad.codes/posts/09-reward-modeling/https://rl.yad.codes/posts/09-reward-modeling/You know how to do RL. But for LLMs, there's a foundational problem: what is the reward for "good" text generation?Wed, 13 May 2026 00:00:00 GMTrlhfhttps://rl.yad.codes/posts/14-constitutional-ai-rlaif/https://rl.yad.codes/posts/14-constitutional-ai-rlaif/Lecture 09 showed how to train a reward model from human preference pairs. Lecture 10 showed how to use that reward model inside PPO to fine-tune a language model. Both assumed a dataset of human-labeled comparisons:…Wed, 13 May 2026 00:00:00 GMTrlaifhttps://rl.yad.codes/posts/15-rl-verifiable-rewards/https://rl.yad.codes/posts/15-rl-verifiable-rewards/Lectures 10–12 covered the standard RLHF pipeline: train a reward model on human preference comparisons, then run PPO against it. The reward model is a learned approximation of "what humans prefer," and its quality…Wed, 13 May 2026 00:00:00 GMTrlvrhttps://rl.yad.codes/posts/16-agentic-rl/https://rl.yad.codes/posts/16-agentic-rl/Lectures 10 through 15 treat the model's output as a single block of text, scored once. You sample a response, a reward model or a verifier returns a scalar, and you update the policy. The "episode" is one generation.Wed, 13 May 2026 00:00:00 GMTagentichttps://rl.yad.codes/posts/17-online-iterative-preference/https://rl.yad.codes/posts/17-online-iterative-preference/Lecture 11: Direct Preference Optimization derived DPO as a one-shot, offline procedure: collect a fixed dataset of (prompt, chosen, rejected) triples, run the DPO loss until convergence, done. No reward model, no…Wed, 13 May 2026 00:00:00 GMTrlhfhttps://rl.yad.codes/posts/18-distillation-reasoning/https://rl.yad.codes/posts/18-distillation-reasoning/Lecture 15: RL with Verifiable Rewards ended with the DeepSeek-R1 recipe: cold-start SFT on a few thousand high-quality reasoning traces, then GRPO against a verifiable reward, then a final alignment pass. The result is…Wed, 13 May 2026 00:00:00 GMTofflinehttps://rl.yad.codes/posts/19-offline-rl/https://rl.yad.codes/posts/19-offline-rl/Lectures 02–08 all assume you can interact with an environment while training. The policy collects transitions, computes gradients, and updates; then does it again. The data distribution changes as the policy improves.…Wed, 13 May 2026 00:00:00 GMTofflinehttps://rl.yad.codes/posts/01-mdps-bellman/https://rl.yad.codes/posts/01-mdps-bellman/Every RL method (Q-learning, PPO, RLHF for LLMs) is solving a Markov decision process. The MDP is the object; the algorithms are different ways of finding a good policy in it. This lecture sets up the object and solves…Wed, 12 Nov 2025 00:00:00 GMTfoundationshttps://rl.yad.codes/posts/02-policy-gradients/https://rl.yad.codes/posts/02-policy-gradients/Policy iteration works on a 5×5 gridworld. It doesn't scale: an LLM's "state" is the whole token sequence so far, so there's no value table big enough, and the action space (the vocabulary) is huge. Policy gradients…Wed, 12 Nov 2025 00:00:00 GMTfoundationshttps://rl.yad.codes/posts/03-value-functions-q-learning/https://rl.yad.codes/posts/03-value-functions-q-learning/Lecture 02 optimized the policy π(a|s) directly. The other route: learn how good each action is (Q(s,a)) and act greedily. That buys you sample efficiency (you can replay old data), off-policy learning (learn from any…Wed, 12 Nov 2025 00:00:00 GMTfoundationshttps://rl.yad.codes/posts/04-actor-critic/https://rl.yad.codes/posts/04-actor-critic/Combine policy gradients with value functions, implement A2C/A3C from scratchWed, 12 Nov 2025 00:00:00 GMTfoundationshttps://rl.yad.codes/posts/05-trpo/https://rl.yad.codes/posts/05-trpo/Understand why policy updates fail, learn TRPO's solution, see why PPO was neededWed, 12 Nov 2025 00:00:00 GMTfoundationshttps://rl.yad.codes/posts/06-ppo/https://rl.yad.codes/posts/06-ppo/PPO (Proximal Policy Optimization) is the standard RL algorithm for RLHF and was used to train InstructGPT and its successors. This lecture covers the algorithm, a full implementation, and practical tuning advice.Wed, 12 Nov 2025 00:00:00 GMTfoundationshttps://rl.yad.codes/posts/07-off-policy-rl/https://rl.yad.codes/posts/07-off-policy-rl/Learn off-policy methods for continuous control, implement SAC and TD3Wed, 12 Nov 2025 00:00:00 GMTclassicalhttps://rl.yad.codes/posts/08-model-based-rl/https://rl.yad.codes/posts/08-model-based-rl/Learn world models, implement Dyna and MBPO, understand when to use model-based RLWed, 12 Nov 2025 00:00:00 GMTclassicalhttps://rl.yad.codes/posts/10-ppo-for-llms/https://rl.yad.codes/posts/10-ppo-for-llms/This lecture covers the three-stage pipeline used in InstructGPT and described in Ouyang et al. (2022, arXiv:2203.02155): supervised fine-tuning, reward model training, then PPO to optimize against that reward model.…Wed, 12 Nov 2025 00:00:00 GMTrlhfhttps://rl.yad.codes/posts/11-dpo/https://rl.yad.codes/posts/11-dpo/PPO-based RLHF works but requires three models (SFT, reward model, policy), has many sensitive hyperparameters, and is expensive to run. DPO (Rafailov et al. 2023, arXiv:2305.18290) shows that you can derive a…Wed, 12 Nov 2025 00:00:00 GMTrlhfhttps://rl.yad.codes/posts/12-beyond-dpo/https://rl.yad.codes/posts/12-beyond-dpo/DPO (2023) eliminated the reward model and PPO instability. But follow-up work exposed weaknesses:Wed, 12 Nov 2025 00:00:00 GMTrlhfhttps://rl.yad.codes/posts/13-rlhf-code-generation/https://rl.yad.codes/posts/13-rlhf-code-generation/Understand how to apply RLHF to code generation, implement CodeRL, and handle execution-based rewardsWed, 12 Nov 2025 00:00:00 GMTrlvr