Study Reinforcement Learning

← home

track eight of eight

Systems

About 7 hours, 6% of the whole. 3 lectures.

learnerrolloutworkersnew weightsrollouts
Figure.The same loop, spread over many GPUs.

How LLM-RL training runs: distributed rollouts and learners, inference engines, hardware.

Start with lecture 29. Each lecture lists what to read first, and ends with a checkpoint: something to do with the page closed.

  1. 29Distributed RL systems2–3 h

    checkpointExplain the IMPALA actor-learner split and what V-trace corrects for; describe a typical 2025 LLM-RL stack (vLLM rollouts + FSDP learner).

    ← start here

  2. 30RL inference infrastructure for LLMs~2 h

    checkpointExplain why decode dominates wall-clock in RL training and what PagedAttention / continuous batching contribute.

  3. 31Hardware for RL~2 h

    checkpointExplain why decode is HBM-bandwidth-bound and prefill is compute-bound; name one kernel-level optimization (FlashAttention) and why it matters.