Why I keep coming back to reinforcement learning

My way into reinforcement learning was a car in GTA V. I wanted it to drive itself.

What I ended up with, in January 2017, was a Keras version of NVIDIA’s end-to-end driving model: a network that looks at a frame and outputs a steering angle. NVIDIA’s paper says theirs learned “with only the human steering angle as the training signal” (Bojarski et al. 2016). Mine learned from a GTA V driving dataset from DeepDrive: recorded frames paired with the steering. So it wasn’t reinforcement learning. It was copying. The car was never told whether a turn went well. It learned to match the recorded steering, frame by frame.

Copying recorded examples, versus learning from what happened when you tried something. That difference is the whole field. So before any history, try it yourself.

Three machines

Three slot machines. Each one pays out some fraction of the time. You don’t know which one is best, and you get 20 pulls.

If you pulled each one a few times and then stuck with whichever looked best, you did roughly what the agent does. It’s called epsilon-greedy. Pull the machine with the best average so far, and one time in ten pull a random one, in case your averages are lying to you. On these machines it averages about 11.5 out of 20. Pulling at random gets about 9.7. Knowing the best machine from the start would get 14.

The gap between 11.5 and 14 is what it costs to find out. Smarter strategies pay less of it. None of them pay nothing.

And notice what you never got. Nobody showed you a labeled example. Nobody told you which machine was right. You got a number back for the machine you pulled, and nothing at all about the two you didn’t. That’s the whole setup of reinforcement learning, and it’s a strange one next to the rest of machine learning.

Labels and rewards

In supervised learning every example comes with the answer. Get it wrong and the loss tells you how wrong, and which way to move.

In RL the reward is usually late. A game of chess is won or lost at the end, not on the move where it was decided. Working out which earlier move deserves the credit is its own problem, credit assignment, and I wouldn’t call it solved.

The smallest version of that I know is the grid below. The +1 is nine steps from the start, and the agent only feels it when it gets there. Press Train 50 episodes and watch the green spread backward from the goal. That’s the agent working out, one update at a time, which earlier squares led to the reward.

Older than deep learning

Most of the core ideas are older than the deep learning boom. Richard Sutton published temporal-difference learning in 1988: update a prediction toward the next prediction instead of waiting for the final outcome. Chris Watkins introduced Q-learning in his 1989 PhD thesis, the algorithm in that grid. Ronald Williams published REINFORCE in 1992, which is still how you take a gradient through a choice you can’t differentiate.

Then the brain showed up. In 1996, Read Montague, Peter Dayan and Terry Sejnowski proposed that dopamine neurons signal something like the TD error, and in 1997 Wolfram Schultz, Dayan and Montague made the case in Science with Schultz’s recordings from monkeys. The neurons fire for a reward nobody predicted, and dip when a predicted reward doesn’t arrive. A learning rule people wrote down for machines matched something brains already do. I don’t know how far to push that. It’s not nothing.

Sutton and Andrew Barto’s textbook came out in 1998. It’s still the book, and the second edition is free online. In March 2025, ACM named the two of them for the 2024 Turing Award, for laying the conceptual and algorithmic foundations of the field.

Inside deep learning

This is the part I think people outside RL miss. RL methods got used as tools inside ordinary deep learning, any time the thing you wanted to optimize had no gradient. Mnih et al. (2014) trained a model to choose where in an image to look next, with REINFORCE, because picking a location isn’t differentiable. Zoph and Le (2017) had a controller propose network architectures and rewarded it with the validation accuracy of what it proposed. Ranzato et al. (2016) and Rennie et al. (2017) trained text generation and captioning directly on the scores they’re judged by, BLEU and CIDEr. AutoAugment searched for data augmentation policies the same way.

And a 2016 note by Pfau and Vinyals points out that GANs and actor-critic methods have the same shape: one network acts, another judges.

Games, then not games

This part you probably know. DeepMind’s DQN learned Atari games from pixels and score, and the 2015 Nature version played 49 games with the same settings. AlphaGo beat Lee Sedol 4 games to 1 in March 2016. AlphaGo Zero learned Go without human games. AlphaZero did the same for chess and shogi, and MuZero learned without being given the rules. In 2019, OpenAI Five beat OG, the reigning Dota 2 world champions, in a restricted version of the game (17 of the 117 heroes), and AlphaStar reached Grandmaster in StarCraft II.

The results I find more convincing are the ones nobody built as a demo. Degrave et al. (2022) shaped and held the plasma in a real tokamak, TCV in Lausanne, with an RL controller. Bellemare et al. (2020) kept Loon’s stratospheric balloons on station the same way, including a 39-day test over the Pacific. AlphaDev found short sorting routines that went into LLVM’s libc++. AlphaTensor found ways to multiply some matrix sizes with fewer multiplications, though its famous 4×4 result only holds in mod-2 arithmetic, and two mathematicians beat one of its records three days later. And back in 2010, Li et al. treated picking front-page stories at Yahoo as a contextual bandit, a grown-up version of the slot machines. On 33 million logged events it got 12.5% more clicks than a bandit that ignored context.

Not every result held up this cleanly. Google’s 2021 Nature paper on laying out chips with RL has been argued over in public since 2023. Independent re-evaluations found the method behind simulated annealing and commercial tools in their setups, Google says those tests didn’t pre-train it or give it enough compute, and the paper was never retracted.

Language models

This is where RL stopped being a niche.

In 2017, Christiano et al. trained agents from human preferences. Show a person two short clips, ask which is better, learn a reward model from the answers, then optimize against it. In 2020, OpenAI used the same recipe to train summarizers. In 2022 came InstructGPT: people preferred the outputs of a 1.3B-parameter model trained this way over the 175B-parameter GPT-3. ChatGPT came out that November, trained with the same method, RLHF. Anthropic’s Constitutional AI replaced much of the human feedback with feedback from a model.

In 2023, DPO showed you could get a lot of the same effect without an RL loop, as a classification loss on pairs of answers. For a while it looked like RL might get optimized out of training language models.

It didn’t. DeepSeekMath introduced GRPO in 2024. OpenAI said o1 was trained with large-scale RL to reason. And in January 2025, DeepSeek-R1 showed that RL on problems with checkable answers, math and code, could teach a model to reason at length. Its R1-Zero variant did it with no supervised fine-tuning first. The agents came next. In 2025 OpenAI said its deep research agent was trained with RL on hard browsing tasks, and that codex-1, the first model behind its Codex coding agent, was trained with RL on real coding tasks, iterating on tests until they pass.

Whether that RL teaches a model anything new is still argued about. Yue et al. (2025) found that base models beat their RL-trained versions when both get many attempts, which suggests RL mostly sharpens what was already there. I don’t think that’s settled.

The gradient underneath GRPO is still the one from Williams’s 1992 paper.

Why it’s worth knowing anyway

My own path went from RL to program synthesis, then NLP with RL, then language models. RL didn’t exactly show up in the program synthesis work. But the loop there (define a DSL, write its grammar, generate a program, check it against an oracle, go back) has the same shape as an RL environment. You propose something, it gets checked, you try again. I don’t know if that counts. It’s close to what RL on checkable answers does to language models now, and I wrote about the synthesis side in Every Coding Agent Trend Is Program Synthesis.

Sutton wrote in 2019 that the methods that win in the long run are the general ones that keep getting better with more compute, which to him means search and learning. He called it The Bitter Lesson. In 2021 he, David Silver, Satinder Singh and Doina Precup argued something bigger, that reward is enough: maximizing reward could account for most of what we call intelligence. I don’t know if that’s right. I’m not sure anyone does yet.

But if you build things with machine learning, sooner or later you’ll want to optimize something you can’t write a loss for. Whether the program passed its tests. Whether the robot stayed upright. That’s reinforcement learning, whatever you end up calling it.

If you want to go through it properly, the 34 lectures here go from MDPs to training language models, and the exercises run from value iteration on a gridworld to GRPO, with tests that only pass when your code is right.

Or maybe I just see RL everywhere because I went looking for it in 2016. That’s possible too.