study reinforcement learning
The red dot starts out knowing nothing about this grid. Press Run and it wanders, falls into the pits, and now and then finds the +1. Every step updates its estimate of how much each move is worth. By fifty episodes it heads straight for the goal.
That's Q-learning. Lecture 03 explains it, and in exercise 03 you write it yourself. From there the guide goes all the way to how language models are trained with RL: 34 lectures in eight tracks, nine exercises with tests, and a checkpoint at the end of every lecture.
Every lecture is studied the same way: read it, watch it, build it, then check yourself with the page closed. Here are the four steps for the Q-learning in Figure 1.
Lecture 03 starts from value functions, builds the update one term at a time, shows that it's the Bellman equation rewritten as an update, and then scales it up to DQN.
Q(s, a) ← Q(s, a) + α [ r + γ maxa′ Q(s′, a′) − Q(s, a) ]
The lecture page links the parts of David Silver's course that cover the same ground: Lecture 4: Model-Free Prediction, Lecture 5: Model Free Control, and Lecture 6: Value Function Approximation. Watch them after reading, or before if you like to see it first.
Open exercise 03. You fill in three TODOs in starter.py, and this is one of them:
def q_update(q_table, state, action, reward, next_state, done, alpha: float, gamma: float = GAMMA) -> float:
"""One Q-learning update, in place on `q_table`.
target = reward if done
reward + gamma * max_a' q_table[next_state, a'] otherwise
q_table[state, action] += alpha * (target - q_table[state, action])
Return the TD error: target - (old q_table[state, action]).
"""
# TODO
raise NotImplementedError("q_update")Then run pytest exercises/03-q-learning/. The five tests pass when the code is right. Stuck? The hints go from a nudge to nearly the answer, and you take them one at a time.
Close the page and answer the lecture's checkpoint:
Explain when you'd reach for a value-based method instead of a policy-gradient one.
Then answer it again a week later, without looking.
Each exercise is a starter file with TODOs, tests that pass only when the code is right, graduated hints, and a reference solution. All nine run on a laptop CPU, from value iteration on a gridworld to GRPO on a toy task with verifiable answers.
Every track builds on Foundations. After that the tracks split by where the reward comes from, and you can take them in the order you need. Each track has its own page, with its lectures in order and the checkpoint for each.
MDPs, value functions, policy gradients, actor-critic, trust regions, PPO. Every other track builds on these.
The environment gives the reward. Off-policy and model-based methods, exploration, multi-agent RL, world models, meta-RL, robotics.
People compare two outputs, a reward model learns from the comparisons, and PPO, DPO, and their relatives optimize against it.
An LLM judge or a written constitution stands in for the human labeler.
A checker decides: the answer matches, the tests pass. Reasoning models, process vs outcome rewards, and how verifiers get gamed.
Multi-step tasks with tools, browsers, and code. The reward usually arrives at the end, so credit assignment gets hard.
Learning from a fixed dataset with no new interaction, including imitation and distillation.
How LLM-RL training runs: distributed rollouts and learners, inference engines, hardware.
Everything in the lecture series is the same underlying object: an MDP, where an agent picks actions and some signal tells it whether things are going well. What changes between sub-fields is mostly what that signal is and who provides it. Classical RL gets a reward from the environment. RLHF infers a reward from human preference labels. RLAIF replaces the human with an LLM judge or a written constitution. RLVR skips the learned reward model entirely and uses a verifier: a checker for math, a test suite for code. Agentic RL puts the model in a multi-turn loop with an environment that tells it whether the task ultimately succeeded. Offline RL works from logged data only, no fresh interaction.
The map below shows where each family fits. The lectures fill in the details; CURRICULUM.md is the suggested order.
Every family below is the same MDP. They differ in three answers: where the reward comes from, where the training data comes from, and how many steps pass before the reward arrives.
env → state → reward → next state. That loop.
All 7 lectures in this track →A learned reward stands in for the environment.
All 5 lectures in this track →Replace the human labeler with an AI.
All 3 lectures in this track →No learned reward model: a verifier is the reward.
All 5 lectures in this track →Long horizons make credit assignment hard.
All 3 lectures in this track →DPO is offline RL on preference data.
All 2 lectures in this track →Underneath all of it: distributed training29, inference30, hardware31.
Reading about RL and getting a policy to learn are different skills. The lectures cover the first. The exercises and checkpoints cover the second, if you use them like this.
starter.py with TODOs and tests that only pass when the code is right. Fill it in before you look at anything else.HINTS.md goes from a nudge to nearly the answer. Read the next hint only when you are stuck on the current one.I'm Yad Konrad. I started this repo on 31 December 2016 as a list of the talks and courses I was learning reinforcement learning from. It grew into the notes, lectures, and exercises on this site.
The first list. Joelle Pineau's and Pieter Abbeel's talks, Sutton and Barto's book, David Silver's course, and Berkeley's CS 294. It is still on the Watch page.
A car that drives itself in GTA San Andreas. That was my way into RL. It became Nvidia-Autopilot-Keras, a Keras version of NVIDIA's end-to-end self-driving model.
Berkeley's CS 294. Deep Reinforcement Learning, taught by Sergey Levine, John Schulman, and Chelsea Finn. My notes from it are kept as I wrote them.
Program synthesis, then NLP with RL, then language models.
The lectures. 34 lectures, exercises with tests, and cheat sheets, drafted with AI coding assistants and then edited. Each lecture stays marked unreviewed until someone has read it end to end and checked it.
This site. The lectures, the exercises, the 2017 notes, and the first list, in one place.