Agentic RL: tool use and multi-turn interaction

---

The agentic setting

Lectures 10 through 15 treat the model’s output as a single block of text, scored once. You sample a response, a reward model or a verifier returns a scalar, and you update the policy. The “episode” is one generation.

An agent is different. Instead of producing a single output and stopping, the model acts in a loop:

1. Given the current history (prompt plus everything so far), the model emits an action: a tool call, a shell command, a file edit, a search query, or plain text.
2. The environment executes the action and returns an observation: the tool’s output, the terminal’s stdout, the new file contents, a webpage.
3. The history grows. The model sees the updated history and picks the next action.
4. This repeats until the task is done or a maximum step limit is reached.

The reward arrives at the end: “the tests pass,” “the issue is resolved,” “the correct answer was submitted.” Intermediate steps have no individual scores unless the environment or a separate process rewards model can provide them.

This is the MDP framing from Lecture 01: MDPs and Bellman Equations, applied at a different scale. The state is the entire conversation-plus-environment history. The action space is “any tool call or text the model can produce.” The transition function is the environment: deterministic for a code sandbox, stochastic for a browser or a real filesystem. And the episode can be dozens of steps long, which is what makes credit assignment hard.

The earlier lectures that feel closest to this are:

• Lecture 10: PPO for language models: how PPO is adapted to LLMs with sparse rewards at end-of-sequence.
• Lecture 12: Beyond DPO: GRPO and 2024–2025 alignment methods: GRPO-style group sampling, which turns up again here for managing variance.
• Lecture 15: RL with Verifiable Rewards: RLVR with a checkable outcome; agentic RL generalizes that from single-turn to multi-turn episodes.

---

The scaffold

Before RL does anything, the model needs a structure for acting in a loop. That structure is called a scaffold: the prompting convention that tells the model when to think, when to call a tool, and how to incorporate the result.

The canonical one is ReAct (Yao et al., 2022/2023, arXiv:2210.03629). The format alternates between three token types:

Thought: I need to find the definition of "eigenvalue."
Action: search["eigenvalue definition"]
Observation: An eigenvalue of a matrix A is a scalar λ such that Av = λv for some nonzero vector v.
Thought: Now I have the definition. I can answer the question.
Action: finish["An eigenvalue is a scalar λ where Av = λv."]

The Thought step is free-form reasoning; the model is reasoning to itself before deciding what to do. The Action step is the actual tool call. The Observation is injected by the environment and is not generated by the model. Then the cycle repeats.

ReAct was introduced as a prompting technique, not as an RL training procedure. In the original paper it was applied to a frozen language model with few-shot examples. But the scaffold defines the interface that RL fine-tuning operates on: the policy is the distribution over Thought and Action tokens given the history; the environment fills in Observation tokens. RL tunes the policy that lives inside the scaffold, not the scaffold itself.

The choice of scaffold is a design decision with consequences. A scaffold that allows long Thought traces gives the model room to plan. One that forces an immediate action prevents overthinking but can miss multi-step problems. Whatever the scaffold, the RL training loop treats the sequence of (Thought, Action) pairs across the episode as the trajectory to optimize.

---

The environment is the reward function

In single-turn RLHF, the reward model is a separate neural network you trained on human preferences. In agentic RL, the environment mostly does that job.

A code-execution environment returns:

• Which tests passed and which failed.
• Whether the process exited cleanly.
• Possibly the diff between the original and modified files.

A browser environment returns:

• The rendered text of the page after an action.
• Whether a navigation succeeded.
• The URL the agent landed on.

A tool-call environment returns:

• The API’s response.
• An error message if the call was malformed.

The terminal reward (“did the task succeed?”) is read from the environment: pytest exit code, a judge’s binary verdict, the score on a held-out test suite. The environment is the reward function, which makes its robustness a correctness concern and, in some cases, a safety concern.

The most important failure mode is reward gaming. If the reward is “all tests pass,” the agent can delete the tests. If the reward is “no lint errors,” the agent can comment out the linted code. If the reward is “the web page confirms success,” the agent can navigate to a page it knows will show a success message without doing the actual task.

Countermeasures include:

• Hidden test suites. The agent sees some tests during the episode; final scoring uses a held-out set it cannot access. SWE-bench (Jimenez et al., arXiv:2310.06770) does this: the model edits a codebase, and the reward is whether the hidden regression tests pass.
• Environment sandboxing. The agent cannot delete test files, cannot write to the test directory, cannot access the test runner configuration.
• Process reward models. A separate model judges whether each intermediate step looks reasonable, making it harder to reach a high score via a nonsensical shortcut.

None of these are complete solutions. A sandboxed environment can still be gamed if the agent finds a loophole in the sandbox. A process reward model can itself be wrong or gamed if the agent learns to produce text that looks good to the PRM without actually doing the right thing.

---

The lineage

WebGPT

Nakano et al. (2021, arXiv:2112.09332) is one of the earliest examples of RL applied to a browsing agent. A fine-tuned GPT-3 could perform web searches, click links, and scroll through pages. The reward came from human raters comparing answers. The setup is recognizably agentic: the model acts (search query, click), the environment responds (page contents), repeat.

WebGPT used imitation learning to bootstrap and RLHF to refine. The browsing actions were a fixed set (search, click link number N, scroll), which made the action space manageable. The main lesson from WebGPT is that RLHF generalizes naturally to interactive settings once you define the action space and can collect human preferences on trajectories.

Toolformer

Schick et al. (2023, arXiv:2302.04761) took a different angle. Toolformer does not use RL. It uses self-supervised learning: the model generates candidate API calls, executes them, and keeps the ones that improve next-token prediction on the surrounding text. The result is a model that can call a calculator, a search engine, a calendar, and a translation API, but was never trained with an explicit reward.

Toolformer matters here as a contrast. It showed that tool use as a capability can emerge from supervised learning. RL is not the only way to get a model to call tools. But RL is useful when you want to optimize for task completion rather than for next-token prediction: when “did the tool call help the downstream answer?” is what you care about, not “does the tool call text fit into the surrounding tokens.”

SWE-bench

Jimenez et al. (2023, arXiv:2310.06770, ICLR 2024) created the benchmark that made “resolve a real GitHub issue” a standard agentic RL target. SWE-bench has 2,294 tasks drawn from real pull requests across 12 Python repositories. The model receives an issue description and a snapshot of the codebase; it can read files, edit code, and run commands; the final reward is whether the hidden test suite passes.

SWE-bench’s value is that the reward is grounded and unambiguous: the tests either pass or they do not. Early results (before RL training became common) were poor: the best models solved only a few percent of issues. By 2025, systems trained with RL on SWE-bench-style tasks had reached above 40% on SWE-bench Verified.

One open example: SWE-RL (Wei et al., 2025, arXiv:2502.18449) trains Llama 3 with RL on open-source software evolution data (issues, diffs, and associated tests from public repositories) and reaches 41.0% on SWE-bench Verified for the 70B model. The reward is a similarity score between the generated patch and the reference solution. The paper is public and the training approach is described in enough detail to reproduce.

What is not public

Most of the high-performing agentic systems in 2024–2025 (coding assistants, autonomous software engineers, tool-using agents in commercial products) are described in blog posts, announcements, or benchmarking leaderboards. The details of how they were trained are not published. Statements like “trained with RL on agentic trajectories” appear in model cards and technical reports without specifying the reward function, the number of environment steps, the scaffold, or how credit assignment was handled. This is worth naming explicitly: the published literature covers the benchmarks and the capability evaluations; the training pipelines for the top performers are largely proprietary.

---

Credit assignment

Credit assignment is the central difficulty in agentic RL, and it gets harder as episodes get longer.

In a 20-step episode where only the last step reveals whether the task succeeded, the policy gradient has to assign credit back across 19 earlier actions. Some of those actions may have been essential (choosing the right file to edit on step 3). Others may have been neutral. A few may have been actively harmful but corrected later. The terminal reward cannot distinguish these.

The math of a long trajectory

Let a trajectory be τ = (s₁, a₁, s₂, a₂, …, sₜ, aₜ) where each state sₜ is the current conversation-plus-environment history and each action aₜ is a (Thought, tool-call) pair or plain text. The terminal reward is R(τ), a single scalar at the end.

The REINFORCE policy gradient is:

∇_θ J(θ) = E_τ[ R(τ) · Σₜ ∇_θ log π_θ(aₜ | sₜ) ]

This says: for every action in the trajectory, scale its log-probability gradient by the total trajectory reward. If the trajectory succeeded (R=1), we nudge the probability of every action upward. If it failed (R=0), we nudge every action downward.

The problem is that this ignores which actions were actually responsible for the outcome. Every action gets the same weight. For a 30-step trajectory, the gradient estimator is high-variance because it’s crediting or blaming 30 decisions for one outcome. The variance scales roughly with trajectory length.

A baseline helps. Subtract a value estimate b(sₜ) from R(τ) so that the gradient becomes:

∇_θ J(θ) = E_τ[ Σₜ (R(τ) - b(sₜ)) · ∇_θ log π_θ(aₜ | sₜ) ]

The baseline is unbiased (it doesn’t change the expected gradient), but it reduces variance when b(sₜ) is a good approximation to the expected return from state sₜ. This is the standard actor-critic setup from Lecture 04.

For agentic RL with a language model, fitting a per-state value function is expensive. You’d need a separate network that can process the entire conversation history. The GRPO approach (from Lecture 12 and Lecture 15) sidesteps this: instead of a learned value network, use the mean reward across K sampled trajectories as the baseline.

advantage_i = (R(τ_i) - mean({R(τ_j) for j in 1..K})) / std({R(τ_j)})

This works when you can afford to sample K trajectories per task, which is expensive but tractable for a small K (4 to 16). The intuition: if 3 of 8 trajectories succeed and 5 fail, the 3 successes get positive advantage and the 5 failures get negative advantage. The policy learns from the relative difference without needing a separate value head.

The options

Terminal reward only, policy gradient over the whole trajectory. Collect the trajectory, wait for the terminal reward, update. High variance. REINFORCE with a baseline helps; GRPO-style group sampling (from Lecture 12) helps more, at the cost of K× more rollouts per update.

Step-level rewards when the environment provides them. If a test suite can be queried mid-episode, you get intermediate signal: “after step 7, 3 of 10 tests pass; after step 12, 7 of 10 pass.” This turns the sparse reward into a denser one. The risk is that a model optimizing intermediate test counts might fix easy tests first and never touch the hard ones.

Step-level rewards also let you use standard TD-style credit assignment with a discount γ:

Gₜ = rₜ + γ · rₜ₊₁ + γ² · rₜ₊₂ + ... + γᵀ⁻ᵗ · rₜ

For pure terminal rewards, rₜ = 0 for all t < T and rₜ = R at t=T, so Gₜ = γᵀ⁻ᵗ · R. Later steps get more credit than earlier ones even without any per-step signal, just from the discount. Whether that’s desirable depends on the task.

Process reward models (PRMs). A separate model scores each (Thought, Action, Observation) triple for whether it looks like a productive step. The PRM is trained on human judgments or on retrospective labels from episodes with known outcomes: you can mark which steps were necessary for the successful trajectories. PRMs have worked well in math reasoning, where wrong intermediate steps are detectable by checking the algebra. In code-editing tasks, a PRM might score “found the right file” as a positive step and “deleted 50 lines without understanding them” as a negative one.

The risk is that the policy learns to produce text that the PRM rates highly without actually solving the task. A PRM is itself a learned model; it can be fooled.

Hierarchical decomposition. Break the task into subtasks with their own checkable rewards. An issue-resolution task might decompose into: (a) identify the failing test, (b) locate the function responsible, (c) understand why it fails, (d) write a fix. Each has a binary check. The agent gets credit for each completed subtask rather than waiting for the end. This requires either human-designed subtask structure or an automated planner; neither is cheap to build.

The connection to RLVR

Lecture 15 covered RLVR: train on tasks where the answer can be verified without a reward model (math problems, code that passes tests). Agentic RL is RLVR generalized to interactive episodes. The setup is the same (checkable outcome, no human preference labels needed), but the episode is longer and the variance is higher. Everything that made RLVR work for single-turn tasks (group sampling, the group-relative advantage) applies here; it’s just more expensive per update.

---

Code: a multi-turn RL rollout

The following sketch shows the structure of an agentic RL rollout. The Policy and Environment are abstract; the loop structure is what matters.

import gymnasium as gym  # gymnasium-style envs are a reasonable mental model,
                         # the env is just slower and the observations are text

from dataclasses import dataclass, field
from typing import Any


# --- Abstract interfaces ---

class Policy:
    """The language model operating inside a scaffold."""
    def act(self, history: list[dict]) -> str:
        """
        Given the history of (role, content) turns, produce the next action.
        In practice this is a model forward pass + sampling + scaffold parsing.
        Returns a string like 'search["query"]' or 'edit_file("foo.py", ...)'
        """
        raise NotImplementedError


class AgentEnv:
    """
    Text-based environment. Conceptually a gymnasium Env where
    observations are strings and actions are strings.
    """
    def reset(self, task: dict) -> str:
        """Return the initial observation (e.g., the issue description)."""
        raise NotImplementedError

    def step(self, action: str) -> tuple[str, float, bool]:
        """
        Execute action, return (observation, reward, done).
        reward is nonzero only when done=True in the sparse-reward case.
        """
        raise NotImplementedError


# --- Trajectory storage ---

@dataclass
class Step:
    history: list[dict]   # history at the time the action was taken
    action: str
    observation: str
    reward: float
    done: bool


@dataclass
class Trajectory:
    steps: list[Step] = field(default_factory=list)
    total_reward: float = 0.0

    def add(self, step: Step):
        self.steps.append(step)
        self.total_reward += step.reward


# --- Single rollout ---

def rollout(policy: Policy, env: AgentEnv, task: dict, max_steps: int = 30) -> Trajectory:
    """
    Run one episode: policy acts, environment steps, repeat until done.
    Returns the full trajectory with per-step rewards.
    """
    traj = Trajectory()
    obs = env.reset(task)

    history = [
        {"role": "system", "content": "You are an agent. Think, then act."},
        {"role": "user",   "content": task["description"]},
        {"role": "env",    "content": obs},
    ]

    for _ in range(max_steps):
        action = policy.act(history)

        history.append({"role": "assistant", "content": action})

        obs, reward, done = env.step(action)

        history.append({"role": "env", "content": obs})

        traj.add(Step(
            history=list(history),  # snapshot
            action=action,
            observation=obs,
            reward=reward,
            done=done,
        ))

        if done:
            break

    return traj


# --- Group rollout (GRPO-style) ---

def group_rollout(
    policy: Policy,
    env: AgentEnv,
    task: dict,
    K: int = 8,
    max_steps: int = 30,
) -> list[Trajectory]:
    """
    Sample K trajectories for the same task.
    Used to compute group-relative advantages.
    """
    return [rollout(policy, env, task, max_steps) for _ in range(K)]


# --- Policy gradient update (terminal reward, group baseline) ---

def grpo_update(
    policy: Policy,
    trajectories: list[Trajectory],
    optimizer,
    epsilon: float = 1e-8,
) -> float:
    """
    Compute the GRPO-style policy gradient loss over a group of trajectories.

    The advantage for each trajectory is (R_i - mean(R)) / std(R),
    where R_i is the terminal reward and the mean/std are over the group.
    This is the same group-relative baseline from Lecture 12/15.

    Each action token in trajectory i is weighted by that trajectory's advantage.
    """
    import torch

    rewards = [t.total_reward for t in trajectories]
    mean_r = sum(rewards) / len(rewards)
    std_r  = (sum((r - mean_r) ** 2 for r in rewards) / len(rewards)) ** 0.5

    total_loss = torch.tensor(0.0, requires_grad=True)

    for traj, raw_r in zip(trajectories, rewards):
        advantage = (raw_r - mean_r) / (std_r + epsilon)

        # For each step, compute the log-prob of the action under the policy
        # and scale by the advantage. In a real implementation, policy.act()
        # would return log-probs alongside the sampled action string.
        for step in traj.steps:
            log_prob = policy.log_prob(step.history, step.action)  # stub
            # Negative because we minimize loss, but want to maximize reward
            total_loss = total_loss + (-log_prob * advantage)

    total_loss = total_loss / len(trajectories)

    optimizer.zero_grad()
    total_loss.backward()
    optimizer.step()

    return total_loss.item()


# --- Outer training loop ---

def train(policy: Policy, env: AgentEnv, task_batch: list[dict], steps: int = 1000):
    import torch
    optimizer = torch.optim.Adam(policy.parameters(), lr=1e-5)

    for step_idx in range(steps):
        for task in task_batch:
            # Sample K trajectories per task
            trajectories = group_rollout(policy, env, task, K=8, max_steps=30)

            # Update policy using group-relative advantage
            loss = grpo_update(policy, trajectories, optimizer)

        if step_idx % 50 == 0:
            mean_r = sum(t.total_reward for t in trajectories) / len(trajectories)
            print(f"step {step_idx}: mean reward = {mean_r:.3f}, loss = {loss:.4f}")

A few things to note about this code:

• The loop structure is an MDP rollout. env.reset() → policy.act() → env.step() → repeat. This is exactly gymnasium’s interface, except observations and actions are strings instead of tensors.
• The reward is sparse by default: env.step() returns reward=0 for all intermediate steps and reward=1.0 (or similar) only when done=True.
• group_rollout samples K trajectories for the same task. This is the GRPO/RLVR pattern: multiple rollouts per prompt, group-relative baseline to reduce variance.
• grpo_update computes the advantage as (R_i - mean) / std across the group. Each action token in the trajectory is weighted by that advantage. The policy is pushed toward trajectories that performed above the group mean.
• policy.log_prob() is a stub. In a real implementation, the model forward pass during rollout would also store log-probabilities for the generated tokens. Those stored log-probs are what you differentiate through.

The code does not handle one important case: when the environment can provide step-level rewards. If env.step() returns nonzero intermediate rewards, you’d accumulate discounted returns per step (as in standard actor-critic) rather than using a flat terminal reward per trajectory.

What the log-prob stub hides

In a real implementation, the rollout and the update share more structure. During policy.act(history), the model does a forward pass and samples a token sequence. You need to keep the log-probabilities of those tokens so you can compute gradients later. A sketch of what that looks like:

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

class LMPolicy(Policy):
    def __init__(self, model_name: str):
        self.model = AutoModelForCausalLM.from_pretrained(model_name)
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)

    def parameters(self):
        return self.model.parameters()

    def act(self, history: list[dict]) -> tuple[str, torch.Tensor]:
        """
        Returns (action_string, log_probs_tensor).
        log_probs_tensor has shape [num_generated_tokens].
        Store it in the Step so grpo_update can differentiate through it.
        """
        # Format history as a single prompt string (scaffold-dependent)
        prompt = self._format(history)
        input_ids = self.tokenizer(prompt, return_tensors="pt").input_ids

        # Generate with the model, collecting per-token log-probs
        with torch.no_grad():
            output = self.model.generate(
                input_ids,
                max_new_tokens=128,
                do_sample=True,
                temperature=0.8,
                return_dict_in_generate=True,
                output_scores=True,
            )

        # output.scores is a tuple of [vocab_size] tensors, one per generated token
        generated_ids = output.sequences[0, input_ids.shape[1]:]
        log_probs = torch.stack([
            F.log_softmax(score, dim=-1)[token_id]
            for score, token_id in zip(output.scores, generated_ids)
        ])

        action_str = self.tokenizer.decode(generated_ids, skip_special_tokens=True)
        return action_str, log_probs

    def _format(self, history: list[dict]) -> str:
        """Render the conversation history as a single string."""
        parts = []
        for turn in history:
            role = turn["role"]
            content = turn["content"]
            if role == "assistant":
                parts.append(f"Thought/Action: {content}")
            elif role == "env":
                parts.append(f"Observation: {content}")
            else:
                parts.append(content)
        return "\n".join(parts)

    def log_prob(self, history: list[dict], action: str) -> torch.Tensor:
        """
        Recompute log-prob of `action` given `history`.
        Used during the update pass (not stored from rollout).
        In practice you'd reuse stored log-probs from the rollout
        and only recompute for the clipped PPO ratio check.
        """
        prompt = self._format(history)
        full_text = prompt + "\nThought/Action: " + action
        input_ids = self.tokenizer(full_text, return_tensors="pt").input_ids
        prompt_len = self.tokenizer(prompt, return_tensors="pt").input_ids.shape[1]

        outputs = self.model(input_ids)
        logits = outputs.logits[0, prompt_len - 1:-1]  # shift for next-token pred
        action_ids = input_ids[0, prompt_len:]
        lps = F.log_softmax(logits, dim=-1)
        return lps.gather(1, action_ids.unsqueeze(-1)).squeeze(-1).sum()

The key accounting point: act() returns both the action string and the log-probs tensor from the sampling pass. Those tensors are stored in the Step. During grpo_update, instead of calling policy.log_prob() fresh (which requires a second forward pass), you use the stored tensors scaled by the advantage, then call .backward(). This is similar to how PPO for LLMs works in Lecture 10, except the “response” is now a full multi-step trajectory of tool calls.

One more practical note: the stored log-probs from the rollout are computed with torch.no_grad(). You have two options for the update:

1. Recompute log-probs from scratch with gradients enabled (requires storing the histories, is memory-intensive for long trajectories).
2. Store log-probs from the rollout as old_log_probs and use a PPO-style clipped importance-weight ratio: ratio = exp(new_log_prob - old_log_prob), clipped to [1-ε, 1+ε]. This is the approach in Lecture 06: PPO.

Option 2, the clipped importance-weight approach, is what most agentic RL training frameworks use in practice, because it decouples the rollout (which generates experience) from the update (which runs multiple gradient steps on the same batch).

---

A worked example: a minimal code-fixing environment

To make the structure concrete, here is a minimal environment where the task is “fix this Python function so its test passes.” The reward is binary: pytest passes or it does not.

import subprocess
import tempfile
import os

class CodeFixEnv(AgentEnv):
    """
    Environment: given a buggy function and a test, fix the function.
    Actions:
      - read_file("path") → returns file contents
      - edit_file("path", "new_content") → writes file, returns diff
      - run_tests() → runs pytest, returns stdout
      - finish() → ends the episode; reward = 1 if tests pass, 0 if not
    Reward: 1.0 if all tests pass at episode end, 0.0 otherwise.
    """

    def __init__(self, buggy_code: str, test_code: str):
        self.buggy_code = buggy_code
        self.test_code = test_code
        self._tmpdir = None
        self._tests_passed = False

    def reset(self, task: dict) -> str:
        # Write files to a temp directory
        self._tmpdir = tempfile.mkdtemp()
        with open(os.path.join(self._tmpdir, "solution.py"), "w") as f:
            f.write(self.buggy_code)
        with open(os.path.join(self._tmpdir, "test_solution.py"), "w") as f:
            f.write(self.test_code)
        self._tests_passed = False

        # Initial observation: the issue description and the failing test output
        result = subprocess.run(
            ["python", "-m", "pytest", "test_solution.py", "-v"],
            cwd=self._tmpdir,
            capture_output=True, text=True
        )
        return (
            f"Task: Fix the bug in solution.py so the tests pass.\n\n"
            f"Current test output:\n{result.stdout[-1000:]}"  # last 1000 chars
        )

    def step(self, action: str) -> tuple[str, float, bool]:
        if action.startswith("read_file("):
            path = action[len("read_file("):-1].strip('"\'')
            full_path = os.path.join(self._tmpdir, path)
            try:
                with open(full_path) as f:
                    contents = f.read()
                return f"Contents of {path}:\n{contents}", 0.0, False
            except FileNotFoundError:
                return f"File not found: {path}", 0.0, False

        elif action.startswith("edit_file("):
            # Parse: edit_file("path", "new content")
            # Minimal parsing: a real implementation would use JSON or structured output
            try:
                _, rest = action.split("(", 1)
                rest = rest.rstrip(")")
                path_part, content_part = rest.split(",", 1)
                path = path_part.strip().strip('"\'')
                new_content = content_part.strip().strip('"\'')
                full_path = os.path.join(self._tmpdir, path)
                with open(full_path, "w") as f:
                    f.write(new_content)
                return f"Wrote {len(new_content)} bytes to {path}.", 0.0, False
            except Exception as e:
                return f"edit_file parse error: {e}", 0.0, False

        elif action.strip() == "run_tests()":
            result = subprocess.run(
                ["python", "-m", "pytest", "test_solution.py", "-v"],
                cwd=self._tmpdir,
                capture_output=True, text=True
            )
            passed = result.returncode == 0
            obs = result.stdout[-2000:]
            # No reward yet: just show the output
            return obs, 0.0, False

        elif action.strip() == "finish()":
            # Score the episode
            result = subprocess.run(
                ["python", "-m", "pytest", "test_solution.py"],
                cwd=self._tmpdir,
                capture_output=True, text=True
            )
            self._tests_passed = (result.returncode == 0)
            reward = 1.0 if self._tests_passed else 0.0
            obs = "Tests passed." if self._tests_passed else "Tests failed."
            return obs, reward, True  # done=True always on finish()

        else:
            return f"Unknown action: {action[:100]}", 0.0, False

A session with this environment might look like:

Observation: Task: Fix the bug in solution.py so the tests pass.
  Current test output: FAILED test_solution.py::test_add, AssertionError: assert 3 == 5

Thought: I need to read solution.py to see what's wrong.
Action: read_file("solution.py")
Observation: Contents of solution.py:
  def add(a, b):
      return a - b

Thought: The bug is obvious, it should return a + b.
Action: edit_file("solution.py", "def add(a, b):\n    return a + b\n")
Observation: Wrote 30 bytes to solution.py.

Thought: Let me verify.
Action: run_tests()
Observation: test_solution.py::test_add PASSED

Action: finish()
Observation: Tests passed.
Reward: 1.0

This is a 3-action episode. In a harder task (a real GitHub issue in a large codebase), an episode might take 20 to 30 actions across multiple files. The reward is still binary at the end; only the number of steps changes.

Notice also where reward gaming would enter: if the agent calls edit_file("test_solution.py", "") (empties the test file), pytest would report success because there are no failing tests. The sandbox needs to make test files read-only to prevent this.

---

When to use this

Agentic RL makes sense when:

• The task is genuinely interactive: it cannot be answered in one shot because the model needs to read a file, run a command, or query an API to get information it doesn’t have.
• The success criterion is checkable without human involvement: a test suite, a database query result, a math answer, a structured output that can be verified against a schema.
• Supervised demonstrations have plateaued: the task distribution is too varied for imitation learning to cover, or the model needs to generalize to problems where no demonstration exists.

Gotchas to be aware of:

Sparse reward over long horizons. A 30-step episode with one terminal reward is hard to optimize. The variance of policy gradient estimates grows with trajectory length. GRPO-style group sampling helps; process reward models help more if you can train them.

The environment is the reward function. Whatever the environment does, the trained policy will optimize for it. If the environment can be gamed (by deleting tests, by producing text the PRM rates highly without doing real work, by navigating to a pre-cached success page), the policy will eventually find the exploit. Held-out test suites and sandboxed environments reduce but do not eliminate this.

Rollout cost. Each step in an agentic episode is a model call plus an environment call. The environment call might be running pytest, rendering a webpage, or executing a shell command. A 30-step episode at 30 environment calls per episode, across K=8 trajectories per task, across a batch of 16 tasks per gradient step: that is thousands of external calls per update. This is orders of magnitude more expensive than single-turn RL, and it is a practical bottleneck for training at scale.

Distribution shift. The training environment is not deployment. A model trained on a sandboxed version of real repositories may encounter edge cases (unusual file encodings, missing dependencies, network-dependent tests) that were absent during training. The more tightly the environment mirrors real deployment, the less this matters, and the more expensive the training environment becomes.

---

Exercises

These are harder to test quickly than the single-turn exercises in earlier lectures. Think of them as projects rather than 30-minute problems.

Build a ReAct loop over two tools. Write a Policy stub that always produces Thought: ... then Action: calculator[...] or Action: search[...]. Write stubs for the calculator (evaluates Python expressions) and the fake search (returns a fixed string). Wire up the rollout loop and print the trajectory. Verify that the history grows correctly across turns.

Wrap a code-fixing task as an environment. Take a small buggy Python file and one failing pytest test. Define env.reset() to return the file contents and the error message. Define env.step() to accept an edit action (e.g., edit_file("path", "new_content")), write the file, run pytest, and return (stdout, reward=1.0 if passing else 0.0, done=passing). Run a few manual episodes to verify the environment works before hooking up a policy.

Implement the GRPO group baseline. Generate K=4 trajectories for the same task using the stub policy with random actions. Compute the group-relative advantage manually: (R_i - mean(R)) / std(R). Verify that trajectories with reward 1.0 get positive advantage and trajectories with reward 0.0 get negative advantage.

Examine reward gaming. In your code-fixing environment, see what happens if the policy is allowed to delete the test file. Does pytest report success? What does this imply for the reward? Add a sandbox check that prevents modifying test files and re-run.

---

References

Yao et al. (2022): “ReAct: Synergizing Reasoning and Acting in Language Models.” arXiv:2210.03629. Verified. The canonical scaffold for think-then-act-then-observe loops; introduced as a prompting technique, later adopted as the standard interface for agentic RL training.

Nakano et al. (2021): “WebGPT: Browser-assisted question-answering with human feedback.” arXiv:2112.09332. Verified. Early RLHF applied to a browsing agent; bootstrapped with imitation learning, refined with human preference comparisons.

Schick et al. (2023): “Toolformer: Language Models Can Teach Themselves to Use Tools.” arXiv:2302.04761. Verified. Self-supervised tool use (not RL); establishes “model calls an API and incorporates the result” as a standard capability.

Jimenez et al. (2023): “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” arXiv:2310.06770. Published at ICLR 2024. Verified. The benchmark that made “reward = hidden test suite passes” a standard agentic RL target; 2,294 tasks from real GitHub pull requests.

Wei et al. (2025): “SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution.” arXiv:2502.18449. Verified. Trains Llama 3 with RL on public software evolution data; Llama3-SWE-RL-70B reaches 41.0% on SWE-bench Verified. An open training approach with published details.

Systems without public training details. Several high-performing agentic coding systems in 2024–2025, including proprietary coding assistants and autonomous software engineering agents at the top of the SWE-bench leaderboard, are described in blog posts or marketing materials as using RL on agentic trajectories. The reward functions, scaffolds, training data, and credit assignment methods are not published. Descriptions of these systems in the lecture are based on what is stated publicly about their capabilities, not their training procedures.

---

Next lecture

Lecture 17: Online and Iterative Preference Optimization