Study Reinforcement Learning

← home

write the code

Exercises

Small coding exercises with pytest tests, graduated hints, and reference solutions. They run on a laptop CPU.

Small, tested coding exercises that go with the lectures. Each one is a NN-topic/ directory:

  • README.md: the task and the acceptance criteria (which tests must pass)
  • starter.py: a skeleton with TODOs; this is the file you edit
  • test_*.py: pytest tests that pass only when your implementation is correct
  • solution/: a reference implementation, for comparing after you’ve tried
  • HINTS.md: graduated hints; hint 1 is a nudge, the last is nearly the answer

Running them

From the repo root:

pip install -r exercises/requirements.txt    # torch, numpy, gymnasium, pytest
pytest exercises/02-policy-gradients/         # run one exercise's tests
pytest exercises/                             # run all of them

Tests that need PyTorch are skipped cleanly if it isn’t installed.

Working with an AI tutor

If you’re using Claude Code or Codex: it should have you edit starter.py, run the tests, and when you’re stuck give you the next hint from HINTS.md, not the solution, not two hints ahead. It should only show you solution/ if you ask for it directly. See ../AGENTS.md. An exercise is done when its tests pass and you can explain why your implementation works.

What’s here

ExerciseGoes withStatus
01-mdps/: value iteration on a gridworldLecture 01ready
02-policy-gradients/: REINFORCE, solve CartPoleLecture 02ready
03-q-learning/: tabular Q-learning on FrozenLakeLecture 03ready
04-actor-critic/: actor-critic with a learned baseline, on CartPoleLecture 04ready
05-ppo/: PPO with GAE, solve CartPoleLecture 06ready
09-reward-model/: Bradley-Terry reward model on synthetic preferencesLecture 09ready
11-dpo/: DPO on a toy preference datasetLecture 11ready
15-grpo-rlvr/: GRPO on a verifiable arithmetic toy taskLecture 15ready
20-exploration/: RND on a sparse-reward chain MDPLecture 20ready

A PPO exercise on a continuous-control env (Pendulum-v1, LunarLanderContinuous-v2) is still the next obvious extension; 05-ppo/ covers the discrete-action case. Agentic RL doesn’t make a fast tested exercise. That’s a “go build it” project.

Source: exercises/README.md on GitHub.