unreviewed draftNo one has checked this end to end. Treat the math, code, and citations as unverified. Source on GitHub.
Goes with Lecture 01. Pairs with exercises/01-mdps/ (the coding part).
What you’ll get out of it
How to write a problem down as an MDP, how to derive the Bellman equations from the definition of value, why value iteration converges (and how fast), and how to actually run it on a small problem.
Prerequisites
- High-school calculus and the idea of expectation. No measure theory needed.
- Read
Sutton & Bartochapter 3 (sections 3.1–3.6) and chapter 4 (sections 4.1–4.4) before starting. They’re short. - Python + NumPy for the coding problem.
Time
About 2.5 hours: ~1.5 h paper, ~45 min code, ~15 min reading.
Files
problems.md: seven problemssolutions.md: full solutions; try the problems first