Lecture 27: RLAIF and synthetic preferences at scale

read first: Lecture 09: Reward modeling for RLHF, Lecture 14: Constitutional AI, RLAIF, and Self-Improvement
checkpoint, for when you finish: Name three biases of an LLM-as-judge and a mitigation for each; explain why a weak judge caps the policy's ceiling.

Where this fits

Lecture 09 walked through reward modeling with human preference pairs. Lecture 14 introduced RLAIF as one of several AI-feedback techniques and sketched the Constitutional AI pipeline. This lecture stays narrow on a single substitution: the labeler is an LLM, not a human. Everything downstream (Bradley-Terry training, PPO, DPO, GRPO) is the same machinery you already know.

Read this after Lecture 14, not instead of it. Lecture 14 covers the overview. This lecture is about what changes when you swap the labeler: the economics, the failure modes, the data scale, and the engineering choices that turn an LLM into a usable preference oracle.

The honest framing is that RLAIF in 2024-2025 is less a research question and more an infrastructure question. The Lee et al. (2023, arXiv:2309.00267) parity result is what made the rest possible; once you accept that AI labels can match human labels on certain tasks, the open problems are about how to generate them cheaply, how to detect when the judge is failing, and how to keep the judge’s quirks from showing up in the policy.

The substitution

The RLHF pipeline you’ve seen is:

prompts -> two responses each -> human labeler -> (chosen, rejected) pairs
        -> reward model trained with Bradley-Terry
        -> policy trained with PPO/DPO/GRPO against the reward model

RLAIF replaces the labeler:

prompts -> two responses each -> LLM judge -> (chosen, rejected) pairs
        -> reward model trained with Bradley-Terry
        -> policy trained with PPO/DPO/GRPO against the reward model

That is the entire change at the diagram level. The full code path for the rest of the system is untouched. If you have a working RLHF stack you can flip it to RLAIF by replacing one function call.

What this hides is that the new function call has a different cost structure, a different noise profile, and a different set of biases than the human labeler it replaced. The rest of the lecture is about those differences.

There is one variant worth naming up front. Lee et al. (2023, arXiv:2309.00267) call the standard pipeline above “canonical RLAIF” or just “RLAIF”. They also introduce “direct RLAIF” (d-RLAIF), which skips the reward model entirely and queries the judge online during RL: every batch of policy rollouts is scored by the judge, and the judge’s output is used directly as the scalar reward. d-RLAIF avoids reward-model overfitting at the cost of paying the judge’s API/compute bill on every training step. We’ll come back to this when discussing OAIF, which generalizes the idea.

The Lee et al. parity result

Lee et al. (2023, arXiv:2309.00267) did the head-to-head experiment the field needed. Same prompts, same base policy, same PPO setup. The only difference: one run used human-labeled preferences from a public dataset, the other used preferences generated by an LLM judge (PaLM 2-L in their setup).

Three tasks: summarization (Reddit TL;DR), helpful dialogue, and harmless dialogue. On all three, human raters shown blind A/B comparisons of RLAIF outputs vs RLHF outputs preferred them at statistically indistinguishable rates. On harmless dialogue, RLAIF actually edged out RLHF: the AI judge applied harmlessness criteria more consistently than the human pool did.

A few things worth knowing about the result:

The judge was a strong, instruction-tuned model. The paper uses PaLM 2-L as the labeler. Replicating with a weak judge (a 1B-parameter model, say) does not produce parity: the agreement rate with human preferences drops, and the trained policy reflects that.
The “human” labels in the comparison are themselves not gold. They’re crowdworker labels on the original datasets, and crowdworker agreement is already only around 70-80% on these tasks. Saying RLAIF “matches” RLHF is partly a statement that AI labels and crowdworker labels are about equally noisy.
“Comparable performance” is measured at one specific data scale (tens of thousands of comparisons). At 10x less data, both methods are worse; at 10x more, the relative ordering can shift. The result is a point estimate, not a function.
They also showed that even when the judge model is smaller than the policy, RLAIF still produces signal. That observation matters for the OAIF online setup, where you can’t always afford a giant judge.

For d-RLAIF specifically, the cost trade is the obvious one: you avoid training a reward model (which can overfit or be distilled away from the true preference) at the cost of paying the judge on every RL step. In their experiments d-RLAIF was at least as good as canonical RLAIF on the tasks studied. Whether it’s worth the inference cost is implementation-specific.

Why the economics change

The thing that makes RLAIF practically interesting is the cost per label.

A human labeler doing a careful preference comparison takes minutes per example and, in the US/EU at production-quality, costs on the order of $1-$3 per pair. OpenAI’s InstructGPT reward-model dataset (33k training prompts, each with 4 to 9 responses ranked by a labeler; Ouyang et al. 2022, arXiv:2203.02155) represents a substantial labor investment. UltraFeedback (Cui et al. 2023, arXiv:2310.01377) is ~64k prompts with multiple rated completions each; a human-labeled equivalent would be infeasible for most academic groups.

An LLM judge using a 7B or 8B open-weights model running on a single GPU costs on the order of $0.001 per pair, three orders of magnitude less than human labeling. A larger judge (Llama 3 70B, or a closed API call) is more expensive but still in the $0.01 range. The numbers vary with hardware, throughput optimization, and prompt length, but the order of magnitude difference holds across reasonable configurations.

The qualitative consequence: dataset sizes that were not feasible become routine. A 1M-comparison dataset costs ~$1k of judge compute instead of ~$1M of human time. That changes what experiments you can run. You can ablate the prompt format. You can re-label after every policy update. You can keep five different judges and use them in an ensemble. You can throw away 90% of the labels and only keep the high-confidence ones.

This is also what enables online AI feedback, which we’ll get to next. If labeling is cheap enough that you can do it during training rather than as a one-shot dataset construction step, the algorithmic options open up.

A blunter way to put the cost story:

Human label:  ~$1.00 per pair  -> 40k pairs ≈ $40k
7B judge:     ~$0.001 per pair -> 40k pairs ≈ $40
70B judge:    ~$0.01 per pair  -> 40k pairs ≈ $400
GPT-4-class:  ~$0.05 per pair  -> 40k pairs ≈ $2000

The numbers above are illustrative: actual costs depend on prompt length, output length, hardware, batching, and whether you’re paying API rates or running on owned hardware. The ratios are what matter for thinking about what’s feasible.

Online AI feedback (OAIF)

Guo et al. (2024, arXiv:2402.04792) push the idea further: don’t label a static dataset at all. Label on-policy, during training, using whatever the current policy produces right now.

The motivation: offline preference data (whether human or AI) is collected once from some reference distribution. Once the policy has drifted away from that distribution, which happens fast in any non-trivial RL run, the offline labels stop being informative about what the policy is currently doing. DPO and standard RLHF both suffer from this; it’s part of why iterative DPO (covered in Lecture 17) and on-policy methods like PPO outperform pure offline DPO.

OAIF closes the loop: at each training step, sample two completions from the current policy, ask the LLM judge which is better, and treat that judgment as the preference signal for the next DPO update. The preference data is generated fresh each step and reflects the current policy’s actual output distribution.

for step in training:
    prompts = sample_batch(prompt_dataset)
    
    # Sample TWO completions per prompt from the current policy.
    completions_a = current_policy.generate(prompts)
    completions_b = current_policy.generate(prompts)
    
    # Judge on-the-fly.
    labels = [
        llm_judge(p, a, b)
        for p, a, b in zip(prompts, completions_a, completions_b)
    ]
    
    # Use the labels as DPO preference pairs.
    current_policy = dpo_step(current_policy, prompts, completions_a, completions_b, labels)

The Guo et al. paper compares this against offline DPO (preferences collected once) and shows OAIF wins on the standard benchmarks they evaluate, mainly because the preference signal stays aligned with the policy’s evolving distribution. It also wins against the d-RLAIF variant in their setup, though the comparison is sensitive to exactly which judge model is used.

OAIF inherits all the LLM-judge biases: position, length, self-preference, the works, but it inherits them in a particular way. Because labeling happens during training, the policy can drift to exploit the judge’s biases if you’re not careful. If the judge prefers slightly longer answers, OAIF will systematically push the policy toward longer answers; iterating this can produce a policy that is dramatically more verbose than the starting point, even though no individual label looked unreasonable. This is the same overoptimization story from Lecture 09: the reward model can be gamed, but now with the AI judge playing the role of the reward model and being gamed in real time.

Mitigations in the paper and in subsequent work: a KL penalty against the reference policy (the same trick as in RLHF), length normalization in the judge’s prompt, periodic re-evaluation against a held-out eval suite, and using a different judge family than the policy family.

UltraFeedback and synthetic preference datasets

Cui et al. (2023, arXiv:2310.01377) released UltraFeedback as a large, open synthetic preference dataset. The headline numbers: ~64k prompts, four completions per prompt sourced from a diverse pool of models (GPT-4, GPT-3.5, Claude, Llama, Mistral, etc.), and per-completion ratings on four dimensions (instruction-following, truthfulness, honesty, helpfulness) produced by GPT-4 acting as a multi-dimensional judge.

A few things make UltraFeedback worth knowing about specifically:

Diverse completion sources. Each prompt has completions from different model families, so the comparisons are not just “same model, two samples”: they span quality and style. This avoids a problem where a single-model preference dataset becomes a calibration set for that model’s idiosyncrasies.
Per-dimension scores, not just preference. Rather than asking “which is better,” the GPT-4 judge gives a 1-5 score on each of the four dimensions. You can build preference pairs from these by comparing aggregate scores, or you can train multi-objective reward models on the per-dimension data.
It became a default substrate. Zephyr (Tunstall et al. 2023, arXiv:2310.16944) was DPO-trained on UltraFeedback-derived pairs and became a reference point for “what you can do with synthetic preferences.” A long list of subsequent open-weight models (Notus, Starling-derived chains, etc.) used UltraFeedback or its derivatives.

The methodology section of the UltraFeedback paper is worth reading even if you never use the dataset, because it documents what choices the authors made when designing the synthetic-labeling pipeline: prompt formatting, instructions to the judge, how they handled judge refusals, calibration runs. A lot of subsequent work has effectively copied this design.

There are a few known issues with the dataset that have come up in follow-up work. The GPT-4 judge has its own biases (covered below). The scores are somewhat compressed: most completions get scores in the 3-5 range, with few outright bad ratings, which can make the preference pairs less informative than they look. And derivative cleaning passes (e.g., the “argilla/distilabel” reprocessing that flagged judge errors) suggest that a non-trivial fraction of the original labels are noisy or wrong on inspection. These are real but they have not stopped the dataset from being broadly useful.

The pattern UltraFeedback set is what gets repeated: pick a strong judge, give it a careful rubric, label completions from a diverse pool of source models, release the dataset, and let the community train against it. Each new generation of strong models (GPT-4-turbo, Claude 3, GPT-4o, etc.) tends to produce a new generation of synthetic preference datasets.

What LLM judges do wrong

Zheng et al. (2023, arXiv:2306.05685) is the foundational paper on LLM-as-judge biases, introduced via the MT-Bench and Chatbot Arena setups. The taxonomy is what later papers have built on. A capable judge can match human preferences at over 80% on well-specified tasks; the open question is what it gets wrong in the remaining 20%, and whether those errors are systematic.

The systematic biases that matter for RLAIF:

Position bias

If the judge sees response A then response B, it tends to prefer A, or sometimes B, depending on the model. The bias is large enough to flip a non-trivial fraction of judgments. Zheng et al. report position-bias rates around 30-50% on closely-matched pairs for several models including GPT-4. If you don’t fight this, you can train a reward model whose main signal is “I am in position A.”

The mitigation is to randomize position per call and, for high-stakes labels, to run each comparison twice with swapped order and only keep judgments that agree. This costs 2x the inference but produces much cleaner data.

Length bias

Longer responses get rated higher, holding content quality roughly constant. This bias is partly real (longer responses sometimes are more thorough) and partly an artifact of how the judge interprets “more comprehensive.” The effect size is large enough that several reward models trained on AI preferences have been shown to mostly encode “longer = better” when probed.

Mitigations: instruct the judge to ignore length; normalize for length explicitly in the reward (subtract alpha * length or train a length-controlled reward head); use length-balanced training data where each chosen response is matched to a rejected response of comparable length; downweight or filter out pairs where the length difference is large.

The instruction-only mitigation works partially. The length-matching mitigation works better but requires you to have, or generate, comparable-length pairs.

Self-preference

A judge rates outputs that match its own style higher than outputs in other styles. GPT-4 rates GPT-4 outputs higher than equivalently-good Claude outputs. Llama rates Llama outputs higher than equivalently-good Mistral outputs. The effect is robust across studies and harder to instruct away than length bias.

The implication for RLAIF: if you use a member of your policy’s family as the judge, you’re training the policy to look like the judge thinks good outputs look like, which can collapse the policy into a mannerism rather than improving quality. The mitigations are to use a different model family as the judge than as the policy, or to use a multi-judge ensemble that spans families.

Verbosity bias

Related to but distinct from length bias: judges prefer responses that pad with explanations, hedges, and meta-commentary. A response that ends “I hope this helps! Let me know if you have any other questions.” may rate higher than the same answer without the closer. This is partly because instruction-tuned judges are themselves trained to be verbose-friendly.

This shows up in the trained policy as a tendency to add filler. It’s particularly bad in long-form generation tasks where the verbose padding dilutes the actual content.

Sycophancy

A judge can be biased by framing in the prompt. If the judge prompt says “the user prefers concise answers,” the judge will rate concise answers higher, even on completions where conciseness is irrelevant or harmful. More subtly, if the judge is given context that one answer was generated by “the better model,” it will rate that answer higher even if the labels are randomized.

This matters for RLAIF in two ways: when constructing judge prompts (don’t leak metadata about which model produced which answer), and when interpreting judge outputs across studies (results that don’t carefully blind the judge are suspect).

Anchoring on the rubric

If you give the judge a rubric, it will mechanically apply that rubric, often to the point of ignoring obvious quality differences that the rubric doesn’t capture. A response that satisfies every bullet point of a checklist can rate higher than a clearly better response that satisfies most of them. This is useful when the rubric is well-designed and a liability when it isn’t.

Calibration drift

Even with the same judge model, the same prompts, and the same instructions, judge outputs can drift across time as the underlying model is updated (for closed-API judges) or as fine-tuning regimes change (for open-weight judges). A reward model trained against gpt-4-0613 is not the same as one trained against gpt-4-1106. This is an ops problem, not a bias problem, but it’s worth knowing.

Mitigations: practical recipes

The biases above are well-known. What people actually do in 2024-2025 to fight them:

Randomize position. Every comparison gets randomized A/B order. For higher-quality data, run each comparison twice with swapped order and only keep agreements. Cost: 2x inference; benefit: cleaner data and a measurable position-bias rate as a side product.

Use pairwise comparisons, not pointwise scores. Pointwise scoring (“rate this 1-5”) is more biased than pairwise comparison (“which is better, A or B”). Bradley-Terry and the rest of the RLHF stack already assume pairwise; you only run into the pointwise problem if you try to use scores as rewards directly. UltraFeedback’s per-dimension pointwise scores are useful for analysis but can be noisy if used as scalar rewards without smoothing.

Use a different model family for the judge than for the policy. If the policy is Llama-based, prefer a non-Llama judge (Mistral, Qwen, or a closed model like GPT-4 or Claude). This won’t eliminate self-preference but it reduces the worst case.

Use multi-judge ensembles. Run the same comparison through 3-5 different judges (different models, different prompts, or both) and take a majority vote or threshold for agreement. Disagreement is itself useful signal: pairs where judges disagree are either genuinely ambiguous (worth less for training) or hard enough that you want a human to look at them.

Filter by judge confidence. If you can extract a confidence score (probability of the chosen label, or a numeric scoring), filter out low-confidence pairs. The remaining data is smaller but cleaner. UltraFeedback’s per-dimension scoring is one way to do this; another is to look at the judge’s log-probabilities on the verdict token.

Length-control the data. Either match chosen and rejected responses to similar lengths before judging, or apply length normalization to the resulting rewards. Both are easier than trying to make the judge length-agnostic via instructions.

Hold out a calibration set. Maintain a small set of human-labeled pairs (~100-500 is enough). Periodically run your judge against this set and report agreement rate. A drop in agreement is a leading indicator that something is wrong: either with the judge, the prompts, or the dataset.

Use principles-based rubrics carefully. The Constitutional AI variant (covered next) gives the judge a written rubric. This can sharpen the signal in specific dimensions but can also create new failure modes when the rubric is incomplete.

Principles-based judging: how the rubric matters

Constitutional AI (Bai et al. 2022, arXiv:2212.08073), covered structurally in Lecture 14, is the canonical “principles-based” judging setup. The judge is not asked “which is better” in the abstract; it’s asked “which response better satisfies this specific principle.” A principle is sampled per comparison; over a dataset, the policy is trained against an aggregate of behaviors implied by the constitution.

This lecture goes further into how the rubric design affects the resulting policy:

Rubric specificity changes what gets optimized. A vague principle (“be helpful”) produces a weak preference signal: the judge falls back on its own priors about helpfulness, which often reduce to length and verbosity biases. A specific principle (“prefer the response that directly answers the user’s question in the first sentence, then provides supporting detail”) produces a much sharper signal but only optimizes for that specific behavior. Aggregate behavior depends on the distribution of principles.

Principles can conflict. “Be concise” and “explain your reasoning thoroughly” are in tension. The constitution as a whole defines a target region in behavior space, not a point. How the model resolves conflicts is implicit in how it interprets the principles, which is itself a function of its pre-training. There is no formal arbitration mechanism.

Rubric order matters. Within a single principle, the order in which sub-criteria are listed influences the judge’s weighting. (“Choose the response that is (a) helpful, (b) accurate, (c) concise” weights helpfulness more than conciseness, even without explicit weights.) This is a small effect, but it accumulates over a dataset.

Principles can be ignored. If a comparison has an obvious quality difference that doesn’t fit the principle, the judge will often go with the obvious difference and ignore the principle. (A clearly harmful response will be ranked lower than a harmless one, even when judged on the principle “prefer the response that is more concise.”) This is mostly a feature: it means principles act as soft filters rather than hard constraints, but it limits what the principles can do.

Multi-dimensional scoring is more informative than scalar preference. Rather than picking the “better” response on a single principle, score each response on a vector of principle-aligned dimensions. UltraFeedback does this. The downstream uses are: training multi-head reward models (one head per dimension), filtering pairs by agreement (use only pairs where the dimensions agree on the winner), or weighting dimensions during training (helpfulness 0.5, harmlessness 0.3, honesty 0.2).

A concrete sketch of a multi-dimensional judge prompt:

You will rate two responses to a user prompt on four dimensions:
  - helpfulness: does it address what the user actually asked?
  - accuracy: is the factual content correct?
  - clarity: is the response well-organized and easy to follow?
  - safety: does it avoid harmful, deceptive, or dangerous content?

For each dimension, give each response a score from 1 to 5.

Prompt: {prompt}
Response A: {response_a}
Response B: {response_b}

Output your scores as JSON:
{{
  "A": {{"helpfulness": ?, "accuracy": ?, "clarity": ?, "safety": ?}},
  "B": {{"helpfulness": ?, "accuracy": ?, "clarity": ?, "safety": ?}}
}}

Parse the JSON, aggregate the scores (sum, weighted sum, or per-dimension), and build preference pairs from the aggregate.

The capability ceiling

The single most important conceptual point about RLAIF: the policy cannot reliably exceed the judge’s evaluative capability on the axes the judge cares about.

If the judge can’t tell that a math proof is wrong, the policy can produce wrong proofs and get high reward. If the judge can’t distinguish good code from code that compiles but doesn’t solve the problem, the policy will learn to produce code that compiles. If the judge has a coarse understanding of “helpfulness” that reduces to “looks long and structured,” the policy will learn to produce long and structured responses regardless of underlying quality.

This is the capability ceiling. It defines the upper bound on what RLAIF can achieve with a given judge.

There are three regimes:

Judge stronger than policy. Common when bootstrapping a smaller policy with a larger judge. The judge can identify mistakes the policy makes and steer it away from them. The policy improves toward the judge’s level. This is the regime where RLAIF works most cleanly.
Judge comparable to policy. What you get in self-rewarding setups (Yuan et al. 2024, arXiv:2401.10020, see Lecture 14). The judge and policy improve together. There’s signal because the judge can identify some of its own mistakes, but the ceiling is fuzzy and the policy can drift into the judge’s blind spots over time.
Judge weaker than policy. What you have to worry about as the policy approaches or exceeds the judge in some dimensions. The policy’s actual improvements stop being recognized by the judge; worse, the policy can learn to exploit the judge’s weaknesses, producing outputs the judge rates highly but that are actually worse. This is the failure mode that limits naive RLAIF on the frontier; once the policy is smarter than the judge in some way, more training in that way actively hurts.

This is the same dynamic that motivates the weak-to-strong generalization line of work. Burns et al. (2023, arXiv:2312.09390), from OpenAI’s superalignment team, frame the problem directly: in the limit, future superhuman models will be evaluated by us (humans), who are weaker. Can a weak supervisor produce a strong model that generalizes correctly?

Their empirical setup: take a strong base model. Have a weak supervisor (a smaller model) generate labels. Fine-tune the strong model on those labels. Measure whether the strong model’s resulting capability exceeds the weak supervisor’s. Their finding: yes, partially. Weak-to-strong generalization happens: the strong model recovers more than the weak supervisor’s level of performance, but there’s still a gap relative to fine-tuning on ground-truth labels. The gap shrinks with techniques like auxiliary confidence losses but doesn’t disappear.

The relevance to RLAIF: a weaker judge can sometimes train a stronger policy, but with reduced efficiency and a real ceiling. RLAIF in 2024-2025 is mostly in regimes 1 and 2, where the ceiling is high enough to be useful. The frontier-model regime (where the policy is or might exceed any available judge in some dimension) is an open research question and partly what motivates evaluations like RewardBench (Lambert et al. 2024, arXiv:2403.13787) that explicitly probe judge quality.

A code sketch: pairwise judge with position randomization and confidence

This is a more careful version of the LLM-judge wrapper from Lecture 14. It does position randomization, double-call agreement, and confidence extraction.

import random
import re
from dataclasses import dataclass

import anthropic

client = anthropic.Anthropic()

JUDGE_MODEL = "claude-haiku-4-5"  # cheap, decent for many tasks

JUDGE_PROMPT = """You are evaluating two responses to a user prompt.
Decide which response is better. Consider helpfulness, accuracy, clarity,
and safety. Ignore differences in length unless one response is substantively
more informative.

[User prompt]
{prompt}

[Response A]
{response_a}

[Response B]
{response_b}

First, briefly note any quality difference (one short sentence).
Then on a new line, output your verdict in this exact format:

VERDICT: A
CONFIDENCE: 0.8

CONFIDENCE is a number between 0.5 (coin flip) and 1.0 (certain).
The VERDICT must be exactly the letter "A" or the letter "B".
"""


@dataclass
class JudgeResult:
    """Output of one pairwise judge call.

    preferred: 0 means response_a was preferred, 1 means response_b. -1 if unparseable.
    confidence: judge-reported confidence in [0.5, 1.0]. None if unparseable.
    raw: the raw text the judge produced, for logging/debugging.
    """
    preferred: int
    confidence: float | None
    raw: str


def _parse_verdict(raw: str) -> tuple[int, float | None]:
    """Parse the VERDICT/CONFIDENCE lines from the judge's output."""
    verdict_match = re.search(r"VERDICT:\s*([AB])", raw)
    if verdict_match is None:
        return -1, None
    preferred = 0 if verdict_match.group(1) == "A" else 1

    confidence: float | None = None
    conf_match = re.search(r"CONFIDENCE:\s*(0?\.\d+|1\.0+|1)", raw)
    if conf_match is not None:
        try:
            confidence = float(conf_match.group(1))
            confidence = max(0.5, min(1.0, confidence))
        except ValueError:
            confidence = None

    return preferred, confidence


def _judge_once(prompt: str, response_a: str, response_b: str) -> JudgeResult:
    """Single judge call, no position randomization."""
    message = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=200,
        messages=[
            {
                "role": "user",
                "content": JUDGE_PROMPT.format(
                    prompt=prompt,
                    response_a=response_a,
                    response_b=response_b,
                ),
            }
        ],
    )
    raw = message.content[0].text
    preferred, confidence = _parse_verdict(raw)
    return JudgeResult(preferred=preferred, confidence=confidence, raw=raw)


def pairwise_judge(
    prompt: str,
    response_a: str,
    response_b: str,
    seed: int | None = None,
) -> JudgeResult:
    """One pairwise judgment with position randomized.

    Internally swaps A and B with 50% probability and flips the verdict back
    so the returned `preferred` is always in the original (a, b) frame.
    """
    rng = random.Random(seed)
    swap = rng.random() < 0.5

    if swap:
        result = _judge_once(prompt, response_b, response_a)
        if result.preferred != -1:
            # Flip preferred back to the original frame.
            result = JudgeResult(
                preferred=1 - result.preferred,
                confidence=result.confidence,
                raw=result.raw,
            )
    else:
        result = _judge_once(prompt, response_a, response_b)

    return result


def pairwise_judge_with_agreement(
    prompt: str,
    response_a: str,
    response_b: str,
) -> JudgeResult:
    """Two judge calls in opposite orders. Returns a verdict only when both agree.

    The returned `preferred` is in the original (a, b) frame. If the two
    orderings disagree, returns preferred=-1 and confidence=None; caller
    should drop this pair from the training set.
    """
    forward = _judge_once(prompt, response_a, response_b)
    reversed_call = _judge_once(prompt, response_b, response_a)

    if forward.preferred == -1 or reversed_call.preferred == -1:
        return JudgeResult(preferred=-1, confidence=None, raw=forward.raw + "\n---\n" + reversed_call.raw)

    # Reversed call: response_b was in position A. Flip its verdict to compare.
    reversed_preferred_in_original_frame = 1 - reversed_call.preferred

    if forward.preferred != reversed_preferred_in_original_frame:
        # Disagreement = unreliable label. Drop the pair.
        return JudgeResult(preferred=-1, confidence=None, raw=forward.raw + "\n---\n" + reversed_call.raw)

    # Agreement. Take the average confidence (or min, depending on how strict you want to be).
    confidences = [c for c in (forward.confidence, reversed_call.confidence) if c is not None]
    avg_conf = sum(confidences) / len(confidences) if confidences else None

    return JudgeResult(
        preferred=forward.preferred,
        confidence=avg_conf,
        raw=forward.raw + "\n---\n" + reversed_call.raw,
    )


def build_preference_pair(
    prompt: str,
    response_a: str,
    response_b: str,
    min_confidence: float = 0.6,
) -> dict | None:
    """Build a (chosen, rejected) pair for downstream Bradley-Terry / DPO training.

    Returns None if the judge was inconsistent or low-confidence; those pairs
    should not enter the training set.
    """
    result = pairwise_judge_with_agreement(prompt, response_a, response_b)

    if result.preferred == -1:
        return None
    if result.confidence is not None and result.confidence < min_confidence:
        return None

    if result.preferred == 0:
        chosen, rejected = response_a, response_b
    else:
        chosen, rejected = response_b, response_a

    return {
        "prompt": prompt,
        "chosen": chosen,
        "rejected": rejected,
        "judge_confidence": result.confidence,
    }

A few notes on this code that are worth making explicit:

The CONFIDENCE field is judge-reported. The judge is often badly calibrated about its own confidence: overconfident on closely-matched pairs, underconfident on obvious ones. Treat it as a useful filter, not a probability.
The “agreement” gate doubles your inference cost. For initial dataset construction this is usually worth it. For OAIF-style online labeling at every training step it may not be.
You can extract a much better confidence estimate by reading the judge’s actual log-probabilities on the verdict token, if you have access (open-weight judges support this directly; some API providers expose top_logprobs). The string-CONFIDENCE shortcut here is what you fall back on otherwise.
The prompt asks for a one-sentence reasoning before the verdict. This is mild chain-of-thought scaffolding and tends to produce more consistent judgments than asking for the letter alone. It does cost more tokens.
The min_confidence filter throws data away. With a strong judge on easy comparisons, you can be aggressive (set 0.8+) and still have plenty of data. On hard comparisons (close pairs, ambiguous tasks) you may have to lower the threshold to get any data through, which is itself a signal that you’re at the edge of the judge’s competence.

Multi-judge ensembles and quorum filtering

The natural extension of the agreement check above is to use multiple judges from different model families. If three independent judges agree on the verdict, that’s stronger evidence than one judge’s double-checked call. If they disagree, the pair is genuinely ambiguous or hard.

A typical setup:

JUDGES = [
    {"provider": "anthropic", "model": "claude-haiku-4-5"},
    {"provider": "openai", "model": "gpt-4o-mini"},
    {"provider": "local", "model": "llama-3.1-70b-instruct"},
]


def quorum_judge(prompt, response_a, response_b, judges=JUDGES, threshold=2):
    """Return a preference only if `threshold` of the judges agree.

    threshold=len(judges) is unanimous; threshold=len(judges)//2 + 1 is majority.
    """
    verdicts = []
    for judge_spec in judges:
        verdict = call_judge(judge_spec, prompt, response_a, response_b)
        if verdict in (0, 1):
            verdicts.append(verdict)

    if not verdicts:
        return -1

    votes_a = verdicts.count(0)
    votes_b = verdicts.count(1)

    if votes_a >= threshold:
        return 0
    if votes_b >= threshold:
        return 1
    return -1  # no quorum

This is the basic shape; real implementations also weight judges (a stronger judge gets a heavier vote), track per-judge agreement rates against a calibration set, and rotate the judge pool to detect drift.

The cost is linear in the number of judges. For dataset construction this is usually fine; for OAIF it’s often too expensive and people drop back to a single judge with agreement checking.

Practical recipes used in 2024-2025

The recipes that have emerged from the dust:

Bootstrap a small policy with a strong judge. Pick a small base policy (1B-8B parameters). Use a frontier model (GPT-4-class, Claude Opus-class) as the judge. Generate synthetic preferences on whatever instruction-following or task data you have. Train with DPO or GRPO. This is the cheap version of UltraFeedback / Zephyr-style training and works well up to the small policy’s capability ceiling.

Self-improve a strong policy with a peer judge. When you can’t get a stronger judge, use a peer model: same size, different family, or same model with different sampling settings. Apply heavy bias mitigation (position randomization, length normalization, multi-judge agreement). Expect modest gains, not transformative ones. The capability ceiling is hard to push past this way.

Use AI eval-loops to verify pipeline regressions. Independent of training, maintain a small LLM-judge eval that grades policy outputs on a held-out prompt set. Run it before and after each training stage. If a stage regresses on this eval, something is wrong: usually a bias the judge introduced. This is cheap insurance.

Multi-stage pipelines. A common shape: SFT on instruction data → DPO on UltraFeedback-style synthetic preferences → optional online RL (OAIF or PPO) for further alignment. The synthetic preference stage is where most of the alignment shaping happens in practice; the online stage refines.

Quorum/agreement-filtered data. Keep only the pairs where multiple judges agree or where one judge agrees with itself across position swaps. Discarding 30-50% of your judges’ outputs to keep the highest-confidence ones is a normal trade: the remaining data is much cleaner and the model trains faster on it.

Periodic human-spot-check. Sample 50-100 pairs from each new synthetic dataset and have someone read them. You’re looking for systematic errors (the judge is misreading the prompt, the responses are mis-formatted in a way that’s confusing the judge, the principle being applied is wrong for the task). This catches problems that automated checks miss.

How RLAIF integrates with DPO, GRPO, and SimPO

Lectures 11 (DPO) and 12 (beyond DPO) showed how to train against preference pairs without an explicit reward model. From RLAIF’s perspective, these are just different consumers of the same preference data:

DPO with synthetic preferences. The most common 2024-2025 recipe. Build a synthetic preference dataset (UltraFeedback-style), train DPO, done. Fast and works well within the data quality bounds.

SimPO with synthetic preferences. SimPO (Meng et al. 2024, arXiv:2405.14734) is a length-normalized variant of DPO that drops the reference model. It often outperforms vanilla DPO on synthetic preference data, partly because the length normalization counteracts one of the standard LLM-judge biases. If you’re using AI labels, SimPO is worth trying alongside DPO.

GRPO with on-the-fly judging. GRPO (Lecture 15) needs a reward signal for each completion. If you’re in a regime where you don’t have verifiable rewards (i.e., not math/code), you can use an LLM judge as the reward function. Sample K completions per prompt, judge them pairwise to construct a ranking, convert the ranking to scalar rewards (e.g., score = rank position), normalize within the group, and run the standard GRPO update. This is computationally expensive (K(K-1)/2 judge calls per prompt) and is mostly used when verifiable rewards aren’t available.

Iterative DPO with AI labels. A common pattern: round 1, DPO on UltraFeedback. Round 2, sample from the round-1 policy, label with a judge, DPO on the new pairs. Round 3, repeat. Each round shrinks the gap between the policy’s distribution and the labeling distribution, which is the same fix as OAIF but in discrete rounds rather than continuously. This is essentially the iterative-DPO recipe from Lecture 17 with the labeler being an LLM rather than a human.

The choice between DPO, GRPO, SimPO, and PPO for the actual policy optimization is mostly independent of whether your preferences are human or AI. The labeling side and the training side are decoupled. What changes is the rate at which you can iterate (cheaper labels mean more rounds) and the failure modes you have to watch for (judge biases bleeding into the trained policy).

A worked end-to-end recipe

To make the integration concrete, here is a stripped-down RLAIF-DPO pipeline. It assumes you have a small instruction-tuned base model and a prompt dataset.

from datasets import Dataset

# 1. Generate two completions per prompt from the current policy.
def generate_completions(policy, prompts, n_per_prompt=2, temperature=0.9):
    completions = []
    for prompt in prompts:
        completions.append([
            policy.generate(prompt, temperature=temperature)
            for _ in range(n_per_prompt)
        ])
    return completions

# 2. Build preference pairs via the judge.
def build_preference_dataset(prompts, completion_pairs):
    pairs = []
    for prompt, (a, b) in zip(prompts, completion_pairs):
        pair = build_preference_pair(prompt, a, b, min_confidence=0.7)
        if pair is not None:
            pairs.append(pair)
    return Dataset.from_list(pairs)

# 3. Train DPO on the synthetic dataset.
def train_dpo_round(policy, preference_dataset, beta=0.1, epochs=1):
    # Standard DPO loop: see Lecture 11 for the loss derivation.
    # The relevant point here is that the dataset format is identical
    # to a human-labeled DPO dataset: {prompt, chosen, rejected}.
    return dpo_train(policy, preference_dataset, beta=beta, epochs=epochs)

# 4. Iterate.
def iterative_rlaif_dpo(initial_policy, prompts, n_rounds=3):
    policy = initial_policy
    for round_idx in range(n_rounds):
        completions = generate_completions(policy, prompts)
        prefs = build_preference_dataset(prompts, completions)
        print(f"Round {round_idx + 1}: kept {len(prefs)}/{len(prompts)} pairs after filtering")
        policy = train_dpo_round(policy, prefs)
        # Run an eval here. If it regresses, stop.
    return policy

A few details that matter in practice:

The temperature when generating completions for judging should be high enough that the two completions actually differ. Temperature 0.7-1.0 is typical. If you sample at temperature 0 you’ll get identical completions and the judge has nothing to compare.
The number of judging calls per prompt determines the dataset size. For n_per_prompt=2 you get one pair per prompt. For n_per_prompt=4 you get up to 6 pairs (4 choose 2) but you’re spending 6x the judge cost per prompt.
Filtering by judge confidence will throw away a fraction of pairs. Track this fraction. A sudden jump in the fraction filtered out is a signal that something has changed (the policy has drifted into a regime the judge is unsure about, for example).
Per-round eval is non-negotiable. The cost of doing eval is small relative to the cost of an unnoticed regression.

When this is the wrong tool

RLAIF is good. It is not the right answer for every alignment problem.

Verifiable tasks should use verifiable rewards. If you’re training a math reasoner, a code generator, or a formal-proof model, use RLVR (Lecture 15). The verifier is more reliable than any judge, and you avoid the judge’s biases entirely. RLAIF on top of RLVR can be useful for combining multiple objectives (correctness + style), but the correctness signal should come from the checker.

Frontier-model alignment is harder than RLAIF can fix. If your policy is at or near the capability frontier in some dimension, the available judges may not be able to distinguish good outputs from outputs that look good. This is the weak-to-strong regime; RLAIF helps but doesn’t suffice, and active research areas like debate, recursive reward modeling, and scalable oversight (Christiano et al. 2018, arXiv:1810.08575) are about closing that gap.

Tasks with subjective per-user values. If “good” depends on the specific user’s preferences (taste in writing style, in jokes, in level of formality), a single judge applies its own preferences uniformly and you lose the per-user signal. You can paper over this by giving the judge a user profile in the prompt, but the result is the judge’s interpretation of the profile, not the user’s actual preference. For genuinely personal-preference tasks, per-user human labels (or per-user fine-tuning) are still better.

Safety-critical evaluation where errors are catastrophic. An LLM judge that gets 95% of safety judgments right is the wrong tool for screening, say, biological-weapons-related outputs. The 5% errors compound across a long training run. Hybrid pipelines with hard-coded filters, human review for flagged cases, and AI judging only for residual decisions are more honest about the failure rate.

When the bias structure of the available judges aligns badly with the task. If the relevant evaluation axis is exactly one that LLM judges are known to be bad at (length-controlled informativeness, say, or detecting subtle factual errors), you may be better off paying for human labels even at high cost.

Exercises

These are doable with a local 7B-or-larger judge model (via Ollama, vLLM, or any API access).

Quantify position bias for one judge model. Take 100 prompt-response pairs where you have ground truth about which response is better (use a small held-out chunk of UltraFeedback or HH-RLHF). Run your judge twice per pair: once with the actual-better response in position A, once in position B. Report the fraction of cases where the judge changes its answer. Hypothesis: the rate is between 5% and 30% for any single-call setup.
Compare judge agreement across model families. Run the same 100 pairs through three different judges (e.g., Mistral 7B, Llama 3 8B, Qwen 2.5 7B). Compute pairwise agreement rates between the judges and the human-labeled ground truth. Compute the rate at which all three agree. Notice that the “all three agree” set has higher agreement with ground truth than any single judge. That’s the case for ensemble filtering.
Measure length bias by intervention. Take 50 pairs of equally-good responses. Append a paragraph of plausible-sounding filler to one of the two responses in each pair. Re-run the judge. Measure how often the judge flips its verdict toward the longer response. If the flip rate is above 20%, your judge has substantial length bias and you should length-normalize before training.
Build a synthetic preference dataset and train DPO. Use the code sketch above (build_preference_pair) to generate 500-1000 preference pairs from a small instruction dataset. Train DPO from a small base (TinyLlama, Pythia 1.4B). Evaluate on AlpacaEval or a simple win-rate eval against the base model. Note how the win-rate looks, and remember that AlpacaEval uses an LLM judge itself, so a model trained against AI preferences will tend to look better than it actually is on AI-judged evals.
Bonus: re-implement OAIF as iterative DPO. Take your trained DPO model from exercise 4. Sample two completions per prompt from the new policy, judge them, build a fresh preference dataset, train another DPO round. Repeat 2-3 times. Compare against running 3 DPO rounds on the same fixed preference dataset. The on-policy iterative version should produce different behavior: likely more drift toward whatever the judge’s biases are. Document what you see.

References

Lee et al. (2023): “RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback.” arXiv:2309.00267. Verified: same paper cited in Lecture 14. The head-to-head RLAIF/RLHF parity result on summarization, helpful dialogue, and harmless dialogue. Introduces direct RLAIF (d-RLAIF) that skips the reward model. Shows even a similar-sized judge can produce useful signal.

Guo et al. (2024): “Direct Language Model Alignment from Online AI Feedback” (OAIF). arXiv:2402.04792. Verified: Feb 2024 arXiv submission, matches ID encoding. Online AI feedback: generate preferences from the current policy on each step rather than from a fixed dataset. Compared against offline DPO and against d-RLAIF; shows on-policy labeling matters for keeping preferences aligned with the evolving policy distribution.

Cui et al. (2023): “UltraFeedback: Boosting Language Models with Scaled AI Feedback.” arXiv:2310.01377. Verified: Oct 2023 submission. Large-scale synthetic preference dataset: ~64k prompts, four completions per prompt from diverse model sources, per-dimension GPT-4 scoring. Became the substrate for Zephyr and many follow-up open models.

Zheng et al. (2023): “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” NeurIPS 2023 (Datasets and Benchmarks Track). arXiv:2306.05685. Verified: same paper cited in Lecture 14. The taxonomy of LLM-judge biases: position bias, verbosity bias, self-preference, length effects. Also documents the ~80% agreement with human preferences for GPT-4 judges that grounds the whole RLAIF program.

Bai et al. (2022): “Constitutional AI: Harmlessness from AI Feedback.” Anthropic. arXiv:2212.08073. Verified: same paper cited in Lecture 14. The principles-based variant of AI feedback. Lecture 14 covers the pipeline; this lecture discusses how the rubric design affects results.

Burns et al. (2023): “Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision.” OpenAI. arXiv:2312.09390. Verified: Dec 2023 submission. Frames the question of whether a weak supervisor can train a strong model. Empirically, partially yes: there’s a generalization gap relative to ground-truth supervision, but the strong model recovers more than the weak supervisor’s level. Directly relevant to the RLAIF capability-ceiling discussion.

Ouyang et al. (2022): “Training language models to follow instructions with human feedback” (InstructGPT). NeurIPS 2022. arXiv:2203.02155. Verified: cited for the ~40k human comparison baseline. The human-RLHF reference point against which RLAIF’s cost story is compared.

Tunstall et al. (2023): “Zephyr: Direct Distillation of LM Alignment.” arXiv:2310.16944. Verified: Oct 2023 submission. DPO trained on UltraFeedback-derived synthetic preferences; a frequently-cited “this works” point for synthetic-preference training of open-weights models.

Meng et al. (2024): “SimPO: Simple Preference Optimization with a Reference-Free Reward.” arXiv:2405.14734. Verified: May 2024 submission. Length-normalized DPO variant that drops the reference model. Mentioned for how it interacts with the length-bias problem in AI labels.

Lambert et al. (2024): “RewardBench: Evaluating Reward Models for Language Modeling.” arXiv:2403.13787. Verified: Mar 2024 submission. Benchmark for evaluating reward models (including LLM judges used as reward models) across categories like chat, safety, and reasoning. Useful for measuring judge quality independently of the policy you’re training.

Recap

Replacing human labelers with LLM judges is the single substitution that makes RLAIF different from RLHF. Lee et al. (2023, arXiv:2309.00267) showed this can match RLHF quality on standard tasks. The cost reduction, roughly three orders of magnitude per label, is what changes the algorithmic options: synthetic preference datasets at millions of pairs, online on-policy labeling (OAIF), multi-judge ensembles, aggressive filtering.

The biases of LLM judges (position, length, self-preference, verbosity, sycophancy) are systematic and propagate into trained policies. Mitigations are well-understood: randomize position, use different families for judge and policy, ensemble multiple judges, filter by agreement, length-normalize, maintain calibration sets. None of these eliminate the biases; they reduce the worst case.

The capability ceiling is the central conceptual constraint. The policy cannot reliably exceed the judge’s evaluative capability on a given axis. When the judge is stronger, RLAIF amplifies capability. When the judge is weaker, RLAIF caps it, or worse, the policy learns to exploit the judge’s weaknesses. Weak-to-strong generalization (Burns et al. 2023, arXiv:2312.09390) is the research line addressing this.

Principles-based judging (Constitutional AI) gives the judge a written rubric and shifts what the policy optimizes for. The rubric design has substantial behavioral effects; the constitution defines a target region in behavior space rather than a target point.

Practically, in 2024-2025: bootstrap small policies with strong judges, use synthetic preferences for the bulk of alignment shaping, layer on online or iterative methods for refinement, and keep human-spot-check loops to detect when something is going wrong.

Next lecture

Lecture 28: Reward hacking and verifier design. Other directions from here: distillation of preference-trained models (extending Lecture 18 to the preference setting), multi-objective RLHF/RLAIF (combining helpfulness, harmlessness, and task-specific rewards), or scalable oversight techniques (debate, recursive reward modeling) for the regime where judges are weaker than policies.

Before moving on, make sure you can:

Describe what changes and what stays the same when swapping a human labeler for an LLM judge.
Quote the order-of-magnitude cost difference and explain what new options it opens up.
List at least four LLM-judge biases and a concrete mitigation for each.
State the capability ceiling claim and explain why it’s the dominant constraint on RLAIF.
Sketch the OAIF training loop and say what problem it’s solving relative to offline DPO.
Explain how the constitution’s rubric shapes what gets optimized.