- read first
- Lecture 06: PPO - Proximal Policy Optimization, Lecture 10: PPO for language models (the full RLHF pipeline)
- checkpoint, for when you finish
- Explain the IMPALA actor-learner split and what V-trace corrects for; describe a typical 2025 LLM-RL stack (vLLM rollouts + FSDP learner).
Why RL training is shaped differently from supervised
Supervised training has a clean separation: there’s a dataset on disk, a loader streams batches to GPUs, and the gradient computation dominates the loop. You can scale by adding more workers reading from the same dataset and synchronizing gradients with AllReduce. The data distribution does not change during training. Once you have a pipeline that fills the GPU at, say, 60% MFU, you mostly just need more GPUs.
RL training breaks every part of that. The data is generated by the policy itself: every gradient step requires fresh rollouts under the current parameters, or you accept that the rollouts are stale and pay for it in algorithmic correctness. The compute splits across two kinds of work: sampling (forward pass only, often with a KV cache, often with temperature > 0) and learning (training-mode forward + backward + optimizer step + activations + gradients + an Adam moment table). These two have different memory footprints, different latency profiles, different optimal batch shapes. A node that’s perfect for one is usually wasteful for the other.
This lecture is about how people actually structure the compute. The first half is the classical pre-LLM distributed RL stack (A3C, IMPALA, Ape-X, SEED RL) because the abstractions they invented (actor/learner split, V-trace correction for policy lag, central inference) carry directly into the modern era. The second half is the LLM-RL stack circa 2024–2025: vLLM/SGLang as rollout backends, FSDP and ZeRO for the learner, NCCL weight broadcast every K steps, reward computed by sandboxed code execution or a separate LLM-judge cluster. The shape is the same; the constants and the bottlenecks moved.
If you only ever train PPO on a single GPU with one environment, none of this matters. The moment you want to scale (more environments, larger policy, or LLM-sized models), you hit the architectural questions immediately.
What “distributed” actually means here
Distributed RL is not one thing. At least three orthogonal axes show up in any system:
- Actor parallelism. Multiple processes running the policy to collect transitions in parallel. Mostly inference. Bottleneck: environment step latency or LLM generation latency.
- Learner parallelism. Gradient computation distributed across GPUs. Bottleneck: gradient AllReduce, parameter sharding overhead, optimizer state memory.
- Replay / buffer infrastructure. Where rollouts live between collection and use. Could be in-memory on the actor, a centralized replay buffer, or a streaming queue.
A vanilla single-machine PPO setup collapses all three onto one process. As you scale, you split them apart, and the interesting questions become:
- How stale can the actor’s policy parameters be before training breaks?
- How do you sync weights from the learner back to the actors? Push? Pull? How often? Full broadcast or delta?
- Where do trajectories live and who pays for serializing them?
- How do you handle preemption: actor crash, learner crash, network partition?
The rest of the lecture walks the history and answers some of these.
The classical actor-learner zoo (2016–2020)
A3C: asynchronous on-policy with parameter copies
Mnih et al. (2016, arXiv:1602.01783) introduced A3C: Asynchronous Advantage Actor-Critic. The idea is straightforward. Run many CPU actor threads, each with its own copy of the network. Each thread interacts with its own environment, computes a local gradient over a short rollout (say, 5–20 steps), and asynchronously applies the gradient to a shared central network. Then it pulls the updated parameters and continues.
Architecturally:
┌─────────────┐
│ Shared θ │
│ (CPU param │
│ server) │
└──┬───────┬──┘
grad apply │ │ grad apply
┌────────┘ └────────┐
▼ ▼
┌────────────┐ ┌────────────┐
│ Actor 1 │ │ Actor N │
│ env step + │ ... │ env step + │
│ local grad │ │ local grad │
└────────────┘ └────────────┘The “asynchronous” part is what made it work on CPUs without GPU coordination. Each actor’s gradient is computed against a slightly stale θ, but the original A3C paper argued that the asynchrony itself helps; decorrelating updates was framed as an alternative to a replay buffer. In practice the algorithm worked well on Atari with 16 CPU threads on one machine, and it was the dominant deep RL recipe for about two years.
What A3C does not solve: it does not handle GPU rollouts cleanly (each thread has a network copy, which wastes GPU memory if you try to put them all there), and the asynchrony makes the policy used to generate any given rollout undefined: it’s “whatever θ was when the actor pulled last.” For short rollouts and small networks this was tolerable. For larger networks and longer rollouts, the policy lag became a real problem.
IMPALA: one learner, many actors, V-trace fixes the lag
Espeholt et al. (2018, arXiv:1802.01561), IMPALA, is the clean version of “one learner consumes trajectories from many actors.” The architecture:
┌─────────────────┐
│ Learner │
│ (GPU node) │
│ forward + back │
│ + optimizer │
└────┬──────┬─────┘
│ │
params ◄────┘ └────► params
push push
│ │
▼ ▼
┌──────────────┐ ┌──────────────┐
│ Actor 1 │ │ Actor M │
│ env step + │ │ env step + │
│ inference │ │ inference │
└──────┬───────┘ └──────┬───────┘
│ trajectories │
└─────────►◄──────┘
│
▼
┌──────────────┐
│ Trajectory │
│ queue │
└──────────────┘
│
▼
LearnerActors do nothing but generate rollouts. They forward-pass the policy, take an environment step, and write the resulting (observation, action, reward, behavior log-prob) tuple onto a queue back to the learner. The learner pulls a batch of trajectories from the queue, runs forward + backward, applies the optimizer update, and periodically pushes new parameters to the actors.
This is purely off-policy in the limit. The actor’s policy μ generated the trajectory; the learner’s policy π is newer and different. Standard on-policy policy gradient stops being unbiased the moment μ ≠ π.
V-trace is the correction. It’s an off-policy actor-critic correction that uses truncated importance sampling ratios. For each timestep t, define the per-step ratio ρ_t = π(a_t|s_t) / μ(a_t|s_t). Then truncate it: ρ̄_t = min(ρ̄_clip, ρ_t). The V-trace target for V(s_t) is:
v_t = V(s_t) + Σ_{k=t}^{t+n-1} γ^{k-t} ( Π_{i=t}^{k-1} c_i ) ρ̄_k ( r_k + γ V(s_{k+1}) - V(s_k) )where c_i = min(c̄_clip, ρ_i) is a second truncated ratio that controls the variance of the correction. The two clipping constants ρ̄_clip (often 1.0) and c̄_clip (often 1.0) play different roles: ρ̄_clip controls the fixed point the value function converges to, and c̄_clip controls how much earlier-timestep correction propagates back.
In words: V-trace lets you train on slightly-stale trajectories without the gradient blowing up. The clip on ρ̄ keeps the policy gradient stable; the clip on c keeps the value target stable. The fixed point you converge to is between the on-policy fixed point (no clipping) and the behavior-policy fixed point (heavy clipping), with the exact location depending on the clip values.
This is the canonical “scale RL by having many cheap actors feed one expensive learner” architecture. Throughput on the original paper: 250k frames/sec on DMLab using ~500 actor CPUs and 1 GPU learner. That ratio, many actors per learner, is the recurring shape of distributed RL.
Ape-X: distributed replay with prioritization
Horgan et al. (2018, arXiv:1803.00933) addressed a different axis. IMPALA is on-policy-with-correction; Ape-X is off-policy with a centralized replay buffer:
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Actor 1 │ │ Actor 2 │ ... │ Actor N │
│ env + │ │ env + │ │ env + │
│ epsilon │ │ epsilon │ │ epsilon │
└────┬─────┘ └────┬─────┘ └────┬─────┘
│ │ │
└──────► ┌──────────────────────────┐
│ Centralized prioritized │
│ replay buffer │
└─────────────┬────────────┘
│ sampled batches
▼
┌──────────────┐
│ Learner │
│ (Q-learning) │
│ + priority │
│ updates │
└──────────────┘Each actor runs its own copy of Q with a different ε for exploration (so the actor population covers a range of exploration rates), pushes transitions to the replay buffer with an initial priority based on TD error, and the learner samples prioritized minibatches and updates priorities after each backward pass. Reported numbers: ~360 actors on Atari, learner throughput around 50k transitions per second consumed.
The contribution beyond “more actors” is that prioritization scales: the buffer is the shared state and the priorities don’t need to be globally synchronized, just per-sample-on-pull. This made it possible to do value-based RL at much larger scale than DQN could reach.
SEED RL: centralize the inference
Espeholt et al. (2019/2020, arXiv:1910.06591), SEED RL, pushes IMPALA’s split in the opposite direction. Instead of each actor running its own inference (which wastes accelerator compute when the actor is on a CPU and the network is GPU-sized), SEED RL puts the inference on the learner accelerator:
Many CPU "actors" running envs
┌─────────────────────────────────────────┐
│ env step → send obs → wait for action │
└────────────────┬────────────────────────┘
│ batched obs (gRPC)
▼
┌──────────────────────────┐
│ Central Inference + │
│ Learner (TPU/GPU) │
│ forward (batched) │
│ forward+back+optim │
└────────────────┬─────────┘
│ actions
▼
back to the actorsThe actors are dumb: they batch up observations from many environments, ship them to the learner, and get actions back. The learner runs a fast batched forward pass for inference and shares the same accelerator for the learner update. This is a much better fit for hardware accelerators because you batch inference across many environments. SEED RL reports throughput improvements over IMPALA of around 1.5–3× for the same hardware budget on the same tasks, and it scales to thousands of envs per accelerator.
The trade-off: every environment step is now a network round-trip. For environments where the step is cheap (Atari, DM Lab), this latency matters and you need fast networking. For environments where the step is expensive (browser, real robot, sandbox), the network round-trip is a small fraction of total time and you barely notice.
Recurrent at scale: R2D2, NGU, Agent57
Three later DeepMind systems extended Ape-X to recurrent policies:
- R2D2: Kapturowski et al., ICLR 2019. Adds LSTM-based agents to the distributed-replay architecture. Two infrastructure problems show up with recurrence and replay: storing the full hidden state for each transition is expensive, and replaying a partial sequence with a wrong initial hidden state (“representational drift”) biases learning. R2D2’s fix is “burn-in”: replay a short prefix of unused states to warm up the LSTM hidden state before the loss computation begins on the rest.
- Never Give Up (NGU): adds intrinsic motivation (episodic and lifelong novelty) for hard-exploration games on top of the R2D2 architecture.
- Agent57: Badia et al. (2020, arXiv:2003.13350). Adds a meta-controller that adaptively chooses between exploratory and exploitative policies. First agent to exceed the human baseline on all 57 Atari games.
The lesson from this line of work: as the algorithm gets more complex, the distributed substrate (Ape-X / R2D2-style) stays approximately the same. Most of the engineering work is in the buffer, the priority updates, and the parameter sync schedule.
Frameworks: Ray/RLlib and Acme
Two open-source frameworks turned these patterns into something you could use without building from scratch.
RLlib: Liang et al. (2018, arXiv:1712.09381). Built on Ray. The abstractions are explicit: a Trainer owns RolloutWorkers (actors) and a Policy (the network). Workers can be local processes, remote Ray actors, or GPU nodes. The framework supports A3C, IMPALA, Ape-X, PPO, SAC, and many others under one API. RLlib’s contribution is that the same code works at one-machine scale and at hundred-machine scale; you change the num_workers config and the framework handles the rest.
Acme: Hoffman et al. (2020, arXiv:2006.00979). DeepMind’s research-oriented framework. More opinionated than RLlib, oriented toward easy reproducibility of distributed RL research. Uses TensorFlow/JAX bindings, integrates with Reverb (a distributed replay buffer) and Launchpad (DeepMind’s job-launching library). Acme tends to be the reference implementation when a DeepMind paper releases code.
If you want to actually do distributed RL today (in the pre-LLM sense), one of these two is where you start. Both are designed around the same actor/learner/replay split that IMPALA and Ape-X formalized.
The on-policy vs off-policy tension, at scale
Single-machine PPO is on-policy. You roll out, you train on those rollouts for a few epochs, you roll out again. The “behavior” policy and the “target” policy are close because the rollouts are fresh.
The moment you split actors and learners across machines, this falls apart. The actor’s parameters are whatever they were when the learner last pushed; the learner is on parameters K steps ahead. Even with frequent pushes, there’s always lag.
A few ways to deal with this:
- Treat the lag as small and ignore it. Works if K is small (a few hundred steps) and the policy doesn’t move much per step. Common in LLM-RL where the learning rate is tiny.
- Account for it with importance sampling. PPO’s clipped ratio
r_t(θ) = π(a_t|s_t) / π_old(a_t|s_t)is already a single-step importance correction. The clipping bounds how much the correction can move the loss. This is enough for moderate lag. - Use V-trace (IMPALA-style). Multi-step truncated importance correction, designed for the actor/learner split.
- Fully off-policy with a replay buffer. Q-learning / SAC / Ape-X. No importance sampling, but you need a value function and the action space has to be manageable.
For LLM-RL the dominant approach is (1) plus (2). Make the lag small by syncing weights frequently, and trust the PPO/GRPO ratio to mop up the residual drift. The clip ratio plus a KL penalty (to a reference, not to the old policy, see Lecture 10) keeps the update conservative even when the rollouts are a few steps stale.
This breaks down once K gets large enough that the importance ratio sits at the clip threshold for most tokens. At that point gradient quality collapses, and your training is doing nothing useful with most of the rollouts. The diagnostic is the clip fraction: what fraction of tokens hit the clip on each update. If it climbs above ~0.5, sync weights more often or reduce the staleness budget.
What changed with LLMs
The LLM era did not invent any new RL algorithm at the systems level (PPO and GRPO were both around). What it changed is the cost profile of the underlying operations:
| Operation | Pre-LLM RL | LLM-RL |
|---|---|---|
| Environment step | µs (Atari) to ms (sim robot) | Seconds to minutes (model generation; tool call; sandbox) |
| Action sampling | Sub-ms forward pass | Many decode steps, each with KV-cache and growing context |
| Policy size | ~10M parameters | 1B–~1T parameters |
| Per-rollout memory | KB | MB to GB (KV cache for long contexts) |
| Gradient step | Few ms | Hundreds of ms to seconds |
| Activations per step | Fit in GPU memory | Need gradient checkpointing, sharding, sometimes offload |
| Reward computation | Constant time (env returns r) | Could be a model forward pass, an LLM judge, or running code |
| Actor-learner sync | A few MB per push | GB to TB per push (full model weights) |
Everything that was cheap got expensive. Everything that was small got large. But the architecture diagram is the same: actors generate, learner updates, weights sync.
The new bottlenecks in order of severity:
- Rollout latency, dominated by decode. A single LLM rollout of 4k tokens at ~50 tokens/sec on a single H100 is 80 seconds. You need either many parallel rollouts, fast inference backends (vLLM, SGLang), or both.
- Weight sync cost. A 70B-parameter model in bf16 is 140 GB. NCCL-broadcasting that from a learner to dozens of actor GPUs is not free. The frequency of sync becomes a hyperparameter.
- Memory pressure on the learner. Adam optimizer state for a 70B model is ~280 GB (8 bytes per param for momentum + variance). The learner needs sharding (ZeRO, FSDP, tensor parallel) just to fit.
- Reward latency when the reward is a model or a code sandbox. The reward computation may itself need a cluster.
The next several sections walk through how the modern stack handles each of these.
The rollout backend: vLLM and SGLang
The cheapest LLM rollout you can write is a HuggingFace generate call with do_sample=True. It also wastes most of your GPU. The reasons:
- The KV cache is allocated per request at maximum length, even if the request finishes early. Memory fragmentation is severe; effective batch size is low.
- Different requests in the same batch have different prompt lengths, so the prefill phase is padded.
- The decode step is one token at a time. Without continuous batching, you wait for the slowest request in the batch to finish before starting the next.
vLLM (Kwon et al. 2023, arXiv:2309.06180) addresses these directly. The headline contribution is PagedAttention: managing the KV cache like an OS manages virtual memory, in fixed-size blocks. No fragmentation, no over-allocation, and the same blocks can be shared across requests with the same prefix. Combined with continuous batching (new requests join the batch as old ones finish), vLLM gets 2–4× throughput over naïve baselines on the same hardware. For RL rollouts, where you generate K completions for many prompts, vLLM is the default.
SGLang (Zheng et al. 2023/2024, arXiv:2312.07104) takes the same idea further with RadixAttention: a radix tree over the KV cache that automatically shares prefixes across many concurrent requests. For RL training, where every rollout in a GRPO group shares the same prompt prefix, this is exactly the right shape. The reported cache hit rates are 50–75% on real workloads. SGLang also has a structured generation language for tool use, which makes it the common choice for agentic RL rollouts.
In an RL training loop, the actor process is essentially “an LLM serving system”: it loads the policy weights, accepts a batch of prompts, generates completions, and returns them. Whether you use vLLM or SGLang depends on local preference; the throughput difference is small compared to the gain over rolling your own.
A typical configuration: each actor node has one or more GPUs running vLLM with tensor parallelism = 1 (if the model fits on one GPU) or = 2/4/8 (if it doesn’t). Multiple actor nodes run in parallel, each pulling prompts from a queue. The learner stays on a separate cluster with a different parallelism strategy entirely.
The learner: ZeRO, FSDP, tensor parallel
The learner has a different problem: it has to hold parameters, gradients, optimizer state, activations, and the forward/backward computation, all on accelerators that can fit maybe 80 GB each. For a 7B model in bf16 the math just barely works on one H100. For 70B or larger, you need to shard.
ZeRO (Rajbhandari et al. 2020, arXiv:1910.02054) is the original sharded-data-parallel scheme. The three stages:
- ZeRO-1: shard the optimizer state across data-parallel ranks. Each rank holds its own slice of Adam moments. Free memory savings, no extra communication beyond what data parallel already does.
- ZeRO-2: also shard the gradients. Each rank reduces gradients for its slice only.
- ZeRO-3: also shard the parameters themselves. Each rank only stores its slice; it AllGathers the full parameter shard on demand for each layer’s forward and backward, then frees it.
ZeRO-3 lets you train models much larger than a single GPU can hold, at the cost of extra AllGather and Reduce-Scatter communication. The cost is amortized when the per-layer computation is large enough.
FSDP (Zhao et al. 2023, arXiv:2304.11277) is PyTorch’s native implementation of ZeRO-3-style sharding. The user wraps each transformer block in FSDP(...), and PyTorch handles the AllGather/Reduce-Scatter automatically. Memory savings are the same as ZeRO-3 in principle. Performance is comparable to DeepSpeed when configured well.
Tensor parallelism is a different axis. Instead of sharding parameters across data-parallel replicas, tensor parallelism splits each weight matrix across the GPUs in a node, so a single forward pass is a parallel computation across (typically) 2/4/8 GPUs. This is necessary when a single transformer block doesn’t fit on one GPU (common for models >70B). Tensor parallel needs high intra-node bandwidth (NVLink); it’s a within-node strategy.
A typical 2025 learner configuration for a 70B model:
- 8 GPUs per node, tensor parallel = 4 within node (so each GPU holds 1/4 of each weight matrix).
- 2 data-parallel groups per node (the other axis of the 8).
- FSDP across all nodes for parameter sharding.
- Gradient checkpointing on every transformer block to save activation memory.
- Mixed precision: bf16 for forward/backward, fp32 master weights for the optimizer.
The combination is sometimes called 3D parallelism (data + tensor + pipeline) or 4D (add sequence parallel). For RL, pipeline parallel is uncommon because the rollout side cares about latency.
Weight sync: the central plumbing problem
Once the actor and learner are on separate hardware, you have to get the learner’s updated weights back to the actor periodically. This is the single most-engineered piece of any LLM-RL stack.
The options:
- Full broadcast every K steps. Learner finishes K gradient updates, then NCCL-broadcasts the full parameter tensor to every actor. Simple. Slow for large models. Adds bandwidth pressure on the inter-cluster link.
- Delta broadcast. Compute the parameter delta since the last sync, broadcast only the delta. Helps if updates are sparse or low-rank, which is sometimes true but often not.
- Checkpoint-based sync. Learner writes a checkpoint to shared storage; actors poll and reload. Simpler ops-wise; latency depends on storage.
- Same-host sync (CUDA IPC or shared memory). If the actor and learner are co-located on the same node, you can avoid the network entirely. Common in smaller setups where one node holds both.
OpenRLHF (Hu et al. 2024, arXiv:2405.11143) uses NCCL to sync between the DeepSpeed-ZeRO learner and the vLLM actor (same physical machine, same NCCL communicator). The HybridFlow / verl paper (Sheng et al. 2024, arXiv:2409.19256) introduces a 3D HybridEngine that explicitly handles the case where the actor and learner have different parallelism strategies (e.g., learner is FSDP+TP, actor is TP only), automatically resharding parameters during the sync.
Sync frequency is a hyperparameter with real consequences. Too frequent: you spend a lot of time on the broadcast and the actors are often idle. Too infrequent: the policy lag grows, the clip ratio saturates, training becomes inefficient. A typical choice is to sync every 1 to a few learner steps, but with many parallel rollouts in flight so the actors stay busy during the sync.
A common trick: double-buffer the actor weights. The actor keeps an old set of weights live for in-flight rollouts and swaps to the new set only when a previous batch finishes. This avoids the actor being entirely idle during sync.
Reward latency
In supervised learning the loss is computed on the GPU in microseconds. In RL, “compute the reward” can take seconds.
Three common reward shapes in LLM-RL and the systems consequences of each:
Learned reward model. A separate transformer (often the same size class as the policy or smaller) takes (prompt, response) and returns a scalar. Computationally similar to one extra forward pass per rollout. Either run on the actor (extra GPU memory) or on a separate reward-model cluster (extra round-trip). Most RLHF stacks do the latter at scale.
Verifiable rule-based reward (math, exact-match, regex). Cheap. Microseconds. Done on CPU after the rollout finishes. This is what makes GRPO + verifiable rewards so fast: the reward computation is essentially free.
Code sandbox reward. Run the generated code through pytest or similar in a sandbox. Latency: hundreds of ms to multiple seconds, depending on test suite. Heavy-tail: some completions hang and need to be killed at a timeout. Often the sandbox cluster is a separate fleet (e.g., a Kubernetes pool of sandboxes accessed via gRPC). Throughput matters: if you do K=8 rollouts per prompt and B=512 prompts per batch, that’s 4096 sandbox runs per training step.
LLM-as-judge reward. Generate a critique with a larger model. Latency: seconds per rollout. Cost: a full LLM forward (often expensive) per completion. Usually batched on a dedicated judge cluster, separate from training.
When the reward is expensive, it becomes the bottleneck. The system design response is to make reward computation asynchronous: rollouts feed into a reward queue, the reward cluster consumes them at its own rate, and the learner only consumes (rollout, reward) pairs that are fully scored. The learner ends up several steps behind the rollouts, which is just one more form of staleness to budget for.
A typical 2025 stack, end to end
Put it all together for a reasoning-model training run, mid-2025 vintage:
┌──────────────────────────────────┐
│ Orchestrator (Ray) │
│ - assigns prompts to actors │
│ - manages queues │
│ - schedules sync windows │
└─────────────┬────────────────────┘
│
┌──────────────────────────┼──────────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌─────────────────┐ ┌──────────────┐
│ Actor pool │ │ Learner cluster │ │ Reward pool │
│ (vLLM nodes) │ │ (FSDP + TP) │ │ (sandboxes, │
│ │ │ │ │ LLM judges) │
│ 32 × 8×H100 │ │ 16 × 8×H100 │ │ 50 × CPU pod │
│ each runs │ │ bf16 train │ │ + 8 × A100 │
│ vLLM TP=2 │ │ fp32 master │ │ for judge │
│ TP=2 instance│ │ ZeRO-3 + TP=4 │ │ │
└──────┬───────┘ └────────┬─────────┘ └──────┬───────┘
│ ▲ │
│ rollouts (text + logp) │ weight sync │ rewards
├─────────────────────────►│ (NCCL bcast every │
│ │ N grad steps) │
│ │ │
▼ │ │
┌──────────────┐ │ │
│ Trajectory │ scored rollouts │ │
│ queue + ref ├──────────────────┘ │
│ log-prob │ │
│ buffer │ ◄───────────────────────────────────────────┘
└──────────────┘The pieces:
- Orchestrator. Often Ray. Owns the prompt queue, decides which actor gets which prompt, manages sync windows.
- Actor pool. Dozens of vLLM-or-SGLang servers, each holding the current policy weights. Generates K completions per prompt. Most of the GPU compute lives here.
- Reward pool. A separate fleet that scores completions. CPUs for sandbox code execution, A100s for an LLM-judge if used.
- Trajectory queue + ref log-prob buffer. After rollouts are scored, the reference policy log-probs are computed (often by the actor as a separate forward pass, or by a dedicated reference inference cluster), and the full record (prompt, completion, reward, behavior log-prob, ref log-prob) goes into a queue.
- Learner cluster. Pulls batches from the queue. Runs the GRPO/PPO update. Periodically broadcasts new weights to the actor pool via NCCL.
Rough numbers for a 70B policy with K=8 rollouts, batch=256 prompts, 4k-token responses:
- Each training step consumes 2048 rollouts × 4k tokens ≈ 8M response tokens.
- vLLM throughput on 8×H100 with TP=2: roughly 6k–8k decode tokens/sec aggregate.
- 32 actor nodes × 4 vLLM instances per node × ~7k tok/sec = ~900k tok/sec aggregate rollout throughput.
- 8M tokens / 900k tok/sec ≈ 9 seconds per batch of rollouts.
- Add ~3–5 seconds for reward (mostly code sandbox), ~2 seconds for weight sync, ~5–8 seconds for the gradient update.
- One full training step in roughly 25–30 seconds.
These numbers are rough order-of-magnitude estimates, not measurements from any specific run. They are here to make the scale concrete, not to be cited as benchmark data. Real numbers depend heavily on prompt length distribution, response length distribution, sandbox latency tail, and the specific hyperparameter K.
Actor-learner loop in pseudo-code
What the orchestration actually looks like at the level of code (heavily simplified: real systems have many more knobs):
# actor.py: runs on each rollout node
import time
class Actor:
def __init__(self, vllm_engine, ref_engine, prompt_queue, rollout_queue):
self.policy = vllm_engine # holds current weights
self.ref = ref_engine # frozen reference, for log-probs
self.prompts_in = prompt_queue
self.rollouts_out = rollout_queue
self.current_step = 0
def maybe_sync_weights(self, learner_step):
# Called by the orchestrator when learner pushes new weights.
# Weights arrive via NCCL broadcast into a CUDA buffer; the engine
# picks them up between batches.
if learner_step > self.current_step:
self.policy.update_weights_from_nccl()
self.current_step = learner_step
def run(self):
while True:
batch = self.prompts_in.get(block=True) # list of prompts, length B
# Generate K completions per prompt with sampling.
completions = self.policy.generate(
prompts=batch.prompts,
n=batch.K,
temperature=1.0,
max_tokens=4096,
)
# completions is shape [B, K], each with text + per-token log-probs
ref_logprobs = self.ref.score(batch.prompts, completions)
self.rollouts_out.put(Rollout(
prompts=batch.prompts,
completions=completions,
behavior_logprobs=completions.logprobs,
ref_logprobs=ref_logprobs,
actor_step=self.current_step,
))# learner.py: runs on the gradient-compute cluster
class Learner:
def __init__(self, model, optimizer, reward_queue, sync_to_actors):
self.model = model # FSDP-wrapped policy
self.opt = optimizer
self.rewards_in = reward_queue # consumes (rollout, reward) pairs
self.broadcast = sync_to_actors # NCCL broadcast handle
self.step = 0
self.sync_every = 4 # push weights every 4 grad steps
def step_once(self):
batch = self.rewards_in.get_batch(min_size=BATCH_PROMPTS)
# batch: B prompts × K completions × (tokens, behavior_lp, ref_lp, reward)
# 1. GRPO advantages: group-relative
rewards = batch.rewards # [B, K]
adv = (rewards - rewards.mean(1, keepdim=True)) \
/ (rewards.std(1, keepdim=True) + 1e-8) # [B, K]
# 2. Recompute current policy log-probs (with grad)
logp_current = self.model.token_logprobs(
batch.tokens, attention_mask=batch.mask
) # [B, K, T]
logp_old = batch.behavior_logprobs.detach()
logp_ref = batch.ref_logprobs.detach()
# 3. PPO-clipped surrogate at the sequence level
ratio = torch.exp(logp_current.sum(-1) - logp_old.sum(-1)) # [B, K]
surr1 = ratio * adv
surr2 = torch.clamp(ratio, 1 - EPS, 1 + EPS) * adv
policy_loss = -torch.min(surr1, surr2).mean()
# 4. KL to reference
kl = (logp_current.sum(-1) - logp_ref.sum(-1)).mean()
loss = policy_loss + BETA * kl
# 5. Standard FSDP backward + optimizer step
loss.backward()
self.opt.step()
self.opt.zero_grad()
self.step += 1
# 6. Sync to actors every N steps
if self.step % self.sync_every == 0:
self.broadcast.push(self.model, step=self.step)
return {"loss": loss.item(), "kl": kl.item(),
"clip_frac": ((ratio - 1).abs() > EPS).float().mean().item()}
def run(self):
while True:
metrics = self.step_once()
log(metrics)A few things in this sketch worth flagging:
- The actor’s policy and the learner’s policy are different objects. The actor runs vLLM (inference-mode, with KV cache). The learner runs the FSDP-wrapped training model. They share weights periodically via NCCL broadcast, but they are different processes with different memory layouts.
logp_oldis detached. It was computed by the actor at rollout time. No gradient flows through it. This is what makes it the “behavior policy” log-prob in the PPO ratio.advis computed withoutmodelat all. It’s a function of rewards only. This is the GRPO trick from Lecture 15: no critic network needed; the baseline is the group mean.broadcast.pushis a hand-wave. The real implementation has to coordinate the actor and learner NCCL groups, handle the parallelism mismatch if any (learner is FSDP+TP, actor might be TP only), and ensure no actor is mid-batch when it swaps weights. The HybridFlow / verl paper describes the resharding step in detail.- There’s no preemption handling. Real systems checkpoint every N steps so a learner crash doesn’t lose everything, and they tolerate actor crashes by reassigning prompts. None of that is in the sketch.
The frameworks that ship this
A few open-source projects that implement most or all of the above stack:
TRL (HuggingFace, huggingface/trl). The most common starter. PPO, DPO, GRPO trainers for transformers, integrated with accelerate. Easy to use at small-to-medium scale. The distributed story is “let accelerate figure it out,” which works for single-node and small multi-node but not for the kinds of asymmetric actor/learner splits described above.
OpenRLHF (Hu et al. 2024, arXiv:2405.11143). Built on Ray + vLLM + DeepSpeed. Explicitly designed for the asymmetric layout: Ray for orchestration, vLLM for rollouts, DeepSpeed-ZeRO for the learner, NCCL for weight sync. Supports PPO, GRPO, DPO, rejection sampling. The framework most directly aligned with the stack diagram above.
DeepSpeed-Chat (Yao et al. 2023, arXiv:2308.01320). Microsoft’s RLHF system. Same ZeRO machinery as DeepSpeed underneath. The original “scale RLHF to large open models” reference implementation; somewhat superseded by newer frameworks but still widely cited.
verl / HybridFlow (Sheng et al. 2024, arXiv:2409.19256). ByteDance’s RLHF framework. The hybrid-controller programming model lets you specify the RL dataflow at a higher level and the system handles the placement and resharding. Integrates with FSDP, Megatron-LM, vLLM, SGLang. Used to train models at the 100B+ scale and reportedly the engine behind several recent open-weight reasoning models.
There’s no single “right” choice. TRL is fine if you fit on a single node. OpenRLHF and verl are reasonable starting points at 8+ GPUs and larger. If you’re going past ~70B parameters, the choice often comes down to which framework supports your sharding strategy and which integrates with your existing training infrastructure.
Specific engineering pain points
A non-exhaustive list of things that go wrong in distributed RL training that you don’t have to deal with in supervised:
Heterogeneous prompts in a batch. Rollouts on prompts of length 100 and length 4000 finish at very different times. Naive batching wastes GPU time on the shorter prompts. Solutions: continuous batching (vLLM/SGLang handle this), sorting prompts into buckets by expected length, dynamic batch reshaping.
The long tail of slow rollouts. A few prompts will produce 8k-token responses while most produce 500. If you wait for the whole batch to finish before moving on, the tail dominates. Solutions: time budgets (kill rollouts at max length), async rollout pipelines (start the next batch before the previous one finishes), more parallel rollouts to amortize the tail.
Actor stragglers. One actor node is slower than the rest (bad GPU, network issue, noisy neighbor). Throughput collapses to the slowest actor. Solutions: redundant rollouts (assign each prompt to multiple actors, take whichever finishes first), straggler detection and reassignment.
NCCL group mismatch. Actor uses TP=2, learner uses FSDP+TP=4. Their NCCL communicators don’t speak the same language. Solutions: explicit resharding during weight sync (HybridFlow), pinning actor parallelism to a strict subset of learner parallelism, or using point-to-point sends instead of collective ops.
Reward signal drift. Reward model is trained on responses from policy v0. After 1000 GRPO steps the policy is at v1, producing distributions the reward model never saw. Reward model becomes unreliable. Solutions: iterative RLHF (retrain the reward model periodically, see Lecture 17) or switch to verifiable rewards where possible (Lecture 15).
Preemption tolerance. Cloud GPUs get preempted. Spot instances disappear without warning. A learner crash without a recent checkpoint is hours of lost training. Solutions: checkpoint every N steps to remote storage, design actors to be stateless (lose an actor → orchestrator reassigns its prompts), keep replay-like buffers durable.
Numerical drift between actor and learner. Actor runs the model in fp16/bf16 with KV cache; learner runs it in bf16 (or higher) for the gradient computation. The log-probabilities can disagree by enough to bias the importance sampling ratio. Solutions: use the same dtype on both sides where possible, or detach + use only the actor’s log-probs as “behavior” log-probs (which is what most frameworks do).
KV cache staleness after weight sync. vLLM caches KV across requests for the same prefix. After a weight sync, those cached KVs are stale (they were computed with the old weights). Most engines invalidate the cache on weight update; the cost is paying the prefill again for prefixes that were previously cached.
When and when not to scale this up
The honest answer for most people learning RL: a single GPU is enough for everything in this lecture series until you cross the 7B-parameter line. If you’re doing GRPO on a 1.5B model with a verifiable reward, you can do it on one H100 with TRL and finish a research-quality experiment in days. None of the asymmetric actor/learner machinery is necessary.
The point at which the asymmetric stack starts to pay off:
- Model too big to fit on one GPU. Once the policy is >~13B parameters, you need some form of sharding for training. The actor can still run on smaller setups if you also have a tensor-parallel inference engine.
- Rollouts are the bottleneck. If your training step is “1 second of gradient + 60 seconds of rollout,” you want many actors in parallel. The learner can sit on a smaller box.
- You’re iterating on reward models / verifiers. The cost of churning through experiments goes down a lot when rollouts are parallelized.
- Reward computation is expensive. Code sandbox or LLM judge means dedicated reward fleets pay for themselves.
The point at which you’re over-engineering:
- Single-node experiments with PPO on Mujoco-sized environments. Use stable-baselines3 or CleanRL. Done.
- DPO / IPO / KTO on preference data. No rollouts during training; supervised-style optimization. Standard data-parallel is fine.
- Bench evaluation runs. You’re not training; you don’t need actor/learner separation.
The general lesson: the distributed RL stack is a tax. Pay it when you have to. The architecture diagrams in this lecture are for the case where you’re already at the point where the single-machine setup is bottlenecking your science.
What’s still hard
Even with all the machinery, several things remain unsolved or actively researched as of late 2025:
Long-horizon agentic rollouts. When a single rollout is a multi-hour browser session or a long coding episode, the storage and reward latency stories both break. Active area; see Lecture 16.
Efficient sync at trillion-parameter scale. NCCL broadcasts of 1T-parameter models are slow even on the best network fabric. People are looking at delta encoding, low-rank approximations, and overlap-with-computation strategies.
Cross-cluster federated RL. Training across multiple datacenters with slower inter-cluster links. Bandwidth-constrained variants of the algorithms above.
Heterogeneous-precision actors. Running the actor at int4 or int8 while the learner is bf16. The importance ratio gets noisier; how to correct for this is open.
Reward model placement and freshness. Should the reward model be co-located with the actor? With the learner? Synced or async? When does retraining the reward model on fresh rollouts pay off?
None of these have a settled answer. The frameworks ship reasonable defaults, but the right call depends on the specific workload.
Recap
Distributed RL has a recurring shape. Many actors generating data, a smaller number of learners updating parameters, some form of buffer or queue between them, and a sync mechanism to keep them connected. A3C established the actor-with-local-copy pattern; IMPALA introduced the actor/learner split with V-trace correction for staleness; Ape-X added prioritized centralized replay; SEED RL centralized the inference on the learner accelerator. RLlib and Acme turned these patterns into reusable frameworks.
The LLM era kept the shape and changed the constants. Rollouts are LLM generations dominated by KV-cache memory and decode latency. Learners use FSDP and ZeRO and tensor parallel to fit models that don’t fit on one GPU. Weight sync between actor and learner is via NCCL broadcast every K steps, with the parallelism strategies of the two sides often differing and the engine handling the resharding. Rewards come from either a learned model (a forward pass), a verifier (microseconds), or a sandbox / judge cluster (seconds). The two main families of open-source frameworks for this are OpenRLHF and verl/HybridFlow; both implement the asymmetric Ray-orchestrated actor/learner/reward stack described above.
The PPO clipped ratio and the KL penalty are what make this work despite the inherent staleness between actor and learner: small lag is absorbed by the clip; large lag breaks training, and the diagnostic is the clip fraction.
Where this connects
- Lecture 06: PPO: the policy gradient method whose ratio + clip is what makes asymmetric actor/learner work despite staleness.
- Lecture 10: PPO for language models: the single-machine version of the LLM-RL loop this lecture distributes.
- Lecture 15: RL with verifiable rewards: GRPO, the algorithm most often run at distributed LLM scale today, and the verifier-as-reward pattern that makes the reward path cheap.
- Lecture 16: Agentic RL: where multi-turn rollouts make the latency story much worse.
- Lecture 17: Online & iterative preference optimization: why the reward model is retrained iteratively, and what that costs in the distributed stack.
References
Mnih et al. (2016): “Asynchronous Methods for Deep Reinforcement Learning.” arXiv:1602.01783. A3C; asynchronous actor-critic with parameter copies. ICML 2016.
Espeholt et al. (2018): “IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures.” arXiv:1802.01561. Central learner, many actors, V-trace off-policy correction.
Horgan et al. (2018): “Distributed Prioritized Experience Replay.” arXiv:1803.00933. Ape-X; distributed actors with a centralized prioritized replay buffer. ICLR 2018.
Kapturowski et al. (2019): “Recurrent Experience Replay in Distributed Reinforcement Learning.” ICLR 2019, OpenReview r1lyTjAqYX. R2D2; distributed replay with LSTM agents and burn-in.
Espeholt et al. (2019/2020): “SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference.” arXiv:1910.06591. Central inference on the learner accelerator. ICLR 2020.
Badia et al. (2020): “Agent57: Outperforming the Atari Human Benchmark.” arXiv:2003.13350. Distributed recurrent agent with adaptive exploration; first to exceed human on all 57 Atari games.
Liang et al. (2018): “RLlib: Abstractions for Distributed Reinforcement Learning.” arXiv:1712.09381. The Ray-based distributed RL framework. ICML 2018.
Hoffman et al. (2020): “Acme: A Research Framework for Distributed Reinforcement Learning.” arXiv:2006.00979. DeepMind’s research framework for distributed RL.
Rajbhandari et al. (2020): “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models.” arXiv:1910.02054. The ZeRO sharding scheme used in DeepSpeed. SC20.
Zhao et al. (2023): “PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel.” arXiv:2304.11277. PyTorch’s native ZeRO-style sharding.
Kwon et al. (2023): “Efficient Memory Management for Large Language Model Serving with PagedAttention.” arXiv:2309.06180. vLLM and PagedAttention. SOSP 2023.
Zheng et al. (2023/2024): “SGLang: Efficient Execution of Structured Language Model Programs.” arXiv:2312.07104. RadixAttention; structured generation for LLM programs. NeurIPS 2024.
Yao et al. (2023): “DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales.” arXiv:2308.01320. Microsoft’s distributed RLHF system.
Hu et al. (2024): “OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework.” arXiv:2405.11143. Ray + vLLM + DeepSpeed RLHF framework.
Sheng et al. (2024): “HybridFlow: A Flexible and Efficient RLHF Framework.” arXiv:2409.19256. The verl framework; hybrid-controller programming model with 3D HybridEngine for actor/learner resharding. EuroSys 2025.
Shao et al. (2024): “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” arXiv:2402.03300. The GRPO algorithm, used at scale in most modern reasoning-model training runs.
HuggingFace TRL: https://github.com/huggingface/trl. The most common entry-point library for PPO/DPO/GRPO on transformers.
Ray: https://github.com/ray-project/ray. The orchestrator most distributed LLM-RL stacks are built on.
vLLM: https://github.com/vllm-project/vllm.
SGLang: https://github.com/sgl-project/sglang.
Debugging notes
Common failure modes when scaling RL training, and what they look like in practice:
-
Clip fraction climbing. If the fraction of tokens whose PPO ratio is outside
[1-eps, 1+eps]rises past ~0.5, the actor’s policy is too stale. Sync weights more often or reduce the rollout-to-update ratio. -
Reward is flat but KL keeps growing. The policy is moving (KL > 0) but not in a direction the reward measures. Often a sign of reward hacking against the learned RM, or that the verifier is poorly scoped. Strengthen the verifier, freeze it more carefully, or raise the KL coefficient.
-
Actor GPUs idle during training step. The actors finished rollouts before the learner finished its step. Either start the next batch of rollouts before sync (async rollout), or push more parallel rollouts so the actor is always busy.
-
Learner OOM on what looks like the same batch shape. Activation memory grew because completions in the batch happened to be longer than usual. Enable gradient checkpointing, lower the maximum completion length, or shard activations across the sequence axis (sequence parallel).
-
NCCL broadcast hangs. One actor crashed or restarted with a different parallelism. The communicator is incomplete. Add timeouts on collective ops; have the orchestrator detect and re-form the NCCL group when an actor goes down.
-
Throughput tanks after first few hundred steps. KV cache fragmentation in the actor: long sessions accumulate cache pressure. vLLM/SGLang handle this; if rolling your own, you’ll see it.
-
All rollouts in a group score identically (zero advantage). GRPO can’t learn from this. The problem is either too easy (all 1.0) or too hard (all 0.0). Curriculum sampling: drop prompts the model has saturated, sample harder ones. Same dynamic as Lecture 15.
If a training run looks fine for the first 100 steps and then collapses, check the weight sync schedule and the clip fraction first. The most common large-scale RL failure is silently growing staleness between actor and learner, which looks like normal training right up until it doesn’t.
Next lecture
Lecture 30: RL inference infrastructure for LLMs: the rollout side of this stack, and why decode dominates the wall-clock time.