Lecture 34: Self-distillation and self-improvement loops

read first: Lecture 11: Direct Preference Optimization (DPO), Lecture 14: Constitutional AI, RLAIF, and Self-Improvement, Lecture 18: Distillation of Reasoning Models
checkpoint, for when you finish: Explain when a self-improvement loop converges vs. collapses; describe the role of the filter / verifier in keeping the loop honest.

Where this fits

Lecture 18: Distillation of Reasoning Models covered a specific recipe: take a large reasoning model, sample reasoning chains, filter by a verifiable checker, train a smaller model on the kept chains. The teacher and student are different models.

This lecture covers the more general pattern, including the case where teacher and student are the same model. A model M generates outputs. Those outputs are filtered or scored by some signal: a verifier, a judge, a heuristic. The filtered set becomes training data for a new version M’. M’ replaces M and the loop repeats.

That’s the whole shape. The variants differ in three places: what signal does the filtering, whether the loop iterates or runs once, and whether the student is the same checkpoint as the teacher or a different model. Lecture 18’s R1-distill recipe is the case where the signal is a verifiable checker, the loop runs once, and teacher and student are different models. Most of the other points in the space have been explored.

The reason the pattern matters: when the filter is reliable, you bootstrap capability without external supervision. The same model that produced the outputs gets trained on its better outputs and improves. There’s no human labeling step. Compute is the only cost. If you have a domain where the filter is trustworthy, and “trustworthy” is the load-bearing word, you can scale capability cheaply.

The same lever cuts the other way when the filter is unreliable. A model trained on its own confident errors gets more confident about those errors. A judge that systematically favors one style produces a student that produces only that style. The loop amplifies whatever signal the filter provides, including its biases.

This lecture is about both sides: when the loop works, when it doesn’t, and the methods that sit along the spectrum from “trustworthy filter, big gains” to “untrustworthy filter, mode collapse.”

Cross-references throughout. The R1-distill specific recipe is Lecture 18, referenced here, not re-explained. Constitutional AI’s RL phase is Lecture 14, same. Iterative DPO and online preference optimization are Lecture 17. The verifiable-rewards setting is Lecture 15.

The general loop

The pattern, written out once:

Given:
  M_0       : initial model
  prompts   : a set of inputs to train against
  filter    : a function that scores or selects outputs
  train     : a training procedure (SFT, DPO, RL)

for t in 0, 1, 2, ...:
    samples_t = [M_t.generate(p) for p in prompts]
    kept_t    = [s for s in samples_t if filter(s) accepts]
    M_{t+1}   = train(M_t, kept_t)

The variants:

Filter type: a binary verifier (math checker, unit tests, theorem prover), a learned reward model, an LLM judge, the model itself acting as judge, a heuristic on the output (length, format, embedding similarity to a reference).
Training procedure: SFT on the kept outputs, DPO on (kept, discarded) pairs, RL with the filter as the reward function. SFT is the most common.
Iteration: one pass (R1-distill), several rounds (STaR, ReST, iterative DPO), or continuous (online preference optimization).
Teacher-student relationship: same model continuing training (self-distillation proper), distinct models (R1-distill), or an ensemble of past checkpoints acting as the teacher.
Prompt source: a fixed labeled dataset, prompts generated by the model itself (Self-Instruct), prompts from an external corpus.

Most named methods are a specific choice on each axis. The methods worth knowing are the ones where the choices are unusual or the failure modes are instructive.

Classical context: self-training before LLMs

The basic idea (train a model, use it to label unlabeled data, retrain on the labels) is decades old. Two reference points:

Yarowsky 1995 (ACL 1995, “Unsupervised word sense disambiguation rivaling supervised methods”) is the canonical NLP application. Train a classifier on a small seed of labeled examples, label unlabeled data with the classifier, keep only the high-confidence labels, retrain. Yarowsky showed this could nearly match supervised performance on word sense disambiguation using almost no manual labels. The mechanism that made it work: the high-confidence threshold filtered out the cases where the classifier was unsure, so only reliable labels entered training.

Lee 2013 (ICML workshop, “Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks”) gave the deep-learning version. During training, predict labels for unlabeled data, treat the argmax as the pseudo-label, include it in the loss. No threshold; the gradient is just smaller on the unlabeled examples via a weighting term. This is even simpler and works on image classification benchmarks.

The two methods bracket the design choice: confidence filtering (Yarowsky) versus down-weighting (Lee). Both are still the moves people use in the LLM era, just with different filters.

The general failure mode in both is the same as in the LLM versions: if the model is confidently wrong on a region of input space, pseudo-labeling reinforces that error. The Yarowsky-style confidence threshold mitigates this by requiring high confidence (which correlates with correctness on well-calibrated models), but a model can be both confident and wrong, especially out of distribution.

Surveys: Triguero, García, Herrera 2015 (“Self-labeled techniques for semi-supervised learning”) in Knowledge and Information Systems is a reasonable older review of self-training in classical ML. For deep semi-supervised learning, Yang, Xu, Liang 2022 (arXiv:2202.13518, “A Survey on Deep Semi-Supervised Learning”) is the modern reference; check it for context, not for LLM-specific guidance.

The conceptual transfer from these older methods to LLM self-distillation is direct. The output space changed from class labels to long sequences of tokens, and the filter changed from “confidence threshold” to “verifiable checker” or “LLM judge,” but the structure is unchanged: model labels data, filter keeps the reliable labels, model retrains on the filtered labels.

STaR: bootstrapping reasoning chains

STaR (Self-Taught Reasoner, Zelikman, Wu, Mu, Goodman 2022, arXiv:2203.14465, NeurIPS 2022) is the LLM-era starting point for self-distillation as a deliberate technique.

The setup: a model that can produce chain-of-thought reasoning, and a dataset of question-answer pairs with known correct answers. The loop:

Prompt the model to produce a chain of thought followed by an answer for each training question.
Check the answer against the ground truth.
Keep the (question, chain, answer) triples where the answer is correct.
SFT the model on the kept triples.
Repeat with the updated model.

The model improves because each round adds correct chains for problems it can solve, and the SFT pushes it to apply the same chain structure to problems it gets wrong. Over rounds, the set of problems the model can solve correctly grows.

The mechanism is consistent with the verifier-driven distillation in Lecture 18. The chain itself is a latent variable: the dataset gives you only (question, answer), and the chain is the missing intermediate explanation. STaR generates candidate chains, uses the answer check as a filter to identify chains that are plausibly correct, and trains on those.

STaR adds one further mechanism: rationalization. For problems the model gets wrong on every sampled chain, STaR generates a chain conditioned on the known correct answer (prompt: “the answer is X; explain how to get there”). These rationalized chains are added to the training set. Without rationalization, the loop’s training set is biased toward problems the model already gets right, easy problems with short chains, and the model never learns the harder structure.

The empirical finding from the paper: STaR’s iterative loop improves arithmetic and commonsense reasoning accuracy over baseline SFT on the same data. The model gets better at producing chains that are both correct and well-structured, even on the harder problems where rationalization was needed.

A pseudocode sketch:

def star_round(model, qa_dataset, num_samples=1):
    """
    One round of STaR.

    For each (question, answer) pair:
      - sample chains from the current model
      - keep chains whose final answer matches the ground truth
      - if no sampled chain is correct, generate a rationalization
        conditioned on the correct answer
    Return the kept (question, chain, answer) triples for SFT.
    """
    training_examples = []

    for question, true_answer in qa_dataset:
        # Step 1: ordinary sampling
        chains = [model.generate_chain(question) for _ in range(num_samples)]
        correct = [c for c in chains if extract_answer(c) == true_answer]

        if correct:
            # Keep the first correct chain (or all of them, depending
            # on the variant). The paper keeps one.
            training_examples.append((question, correct[0]))
            continue

        # Step 2: rationalization fallback
        rationalized = model.generate_chain_given_answer(question, true_answer)
        if extract_answer(rationalized) == true_answer:
            training_examples.append((question, rationalized))
        # If even rationalization fails, drop this example for this round.

    return training_examples


def star(initial_model, qa_dataset, num_rounds=5):
    model = initial_model
    for r in range(num_rounds):
        examples = star_round(model, qa_dataset)
        print(f"Round {r}: kept {len(examples)} / {len(qa_dataset)} examples")
        model = sft(model, examples, epochs=2)
    return model

The relationship to expectation-maximization is explicit in the STaR paper and worth being precise about. The latent variable is the chain. The E step samples candidate chains from the current model and keeps the ones consistent with the observed answer (the filter). The M step does SFT on the kept chains, which is approximately maximizing the log-likelihood of the data under the model with the chain treated as observed. This isn’t EM in the strict statistical sense (the filter is a hard threshold rather than a posterior weighting, and the M step is one or two epochs of SGD rather than a full likelihood maximization), but the structure is the same.

The rationalization step doesn’t fit the EM framing cleanly. It’s an importance-sampling-style fix for the case where the current model assigns near-zero probability to any chain that reaches the correct answer. Without rationalization, those questions contribute nothing to the loss and the model stays stuck. With rationalization, the loop bootstraps onto problems the model couldn’t initially solve.

ReST and ReST^EM: explicit EM framing

ReST (Reinforced Self-Training, Gulcehre, Le Paine, Srinivasan et al. 2023, arXiv:2308.08998) restructures STaR-style self-training for compute efficiency.

The structural difference: STaR interleaves generation and training within each round. For each batch of questions, sample, filter, train, move on. ReST separates them. The “Grow” step generates a large batch of completions from the current model all at once, in parallel, on whatever inference hardware is available. The “Improve” step then runs several rounds of training on the resulting filtered dataset, with the filtering threshold raised between training rounds.

Grow step:
  Generate K completions per prompt from current policy
  Score each completion with the reward model or verifier

Improve step (repeated several times):
  Filter completions to those above reward threshold T_i
  Train on filtered completions
  Increase threshold: T_{i+1} > T_i
  (Training set shrinks each Improve iteration as threshold rises)

After all Improve iterations: optionally Grow again with new policy.

The structural advantages:

Inference parallelism: generation is embarrassingly parallel. Doing it in one bulk Grow step uses inference hardware more efficiently than interleaving.
Threshold escalation: starting with a low threshold and raising it lets the model first learn from many merely-acceptable outputs, then specialize to the best. This is similar in spirit to curriculum learning.
No training-time reward computation: scoring happens once at Grow time, not on the fly during training. Cheaper if the reward model is large.

The trade-off is data freshness. By the time the third Improve iteration runs, the model is several updates away from the policy that generated the data. That’s stale by the criteria of Lecture 17: on-policy preference data is better-calibrated than off-policy preference data. ReST is on-policy at the start of each outer Grow loop and progressively drifts off-policy through the inner Improve loop.

ReST^EM (Singh, Co-Reyes, Agarwal et al. 2023, arXiv:2312.06585, “Beyond Human Data”) makes the EM connection explicit. The same Grow/Improve structure, framed as approximate EM:

E step (Grow): sample candidate completions from the current policy. The latent variable is the completion. The “observation” is the binary reward (1 if the completion satisfies the filter, 0 otherwise). Filtering to correct completions approximates the posterior P(completion | observation=1).
M step (Improve): maximize the log-likelihood of the filtered completions under the model. This is the SFT step.

The paper’s empirical finding: on math and code, ReST^EM outperforms standard SFT on the same dataset by a significant margin, and the gap grows with iteration count. The interpretation is consistent with the general loop intuition: each iteration adds correct completions for harder problems, expanding the training distribution.

ReST^EM also explicitly compares to training on human-written solutions for the same problems. The model-generated correct chains, filtered by the verifier, sometimes outperform human-written chains as training data. This is striking; it suggests that for math reasoning specifically, a strong model’s own correct chains are at least as informative as human-written ones, because they’re tailored to the model’s existing reasoning style.

The connection back to Lecture 18: R1-distill is essentially one round of cross-model ReST^EM where the Grow model (R1) is much stronger than the Improve target (Qwen-7B). The single-round version works well when the gap between Grow model and Improve target is large; the iterative version is necessary when the two are the same model and have to improve together.

Self-Instruct: bootstrapping instructions, not just answers

Self-Instruct (Wang, Kordi, Mishra et al. 2022, arXiv:2212.10560, ACL 2023) does the loop one level higher: instead of bootstrapping correct outputs to fixed instructions, it bootstraps the instructions themselves.

The setup: a small seed set of 175 hand-written task instructions, each with an example input and output. The model uses this seed to generate new instructions, then generates inputs and outputs for those new instructions. The generated set is filtered for diversity and basic quality (deduplication, exclusion of malformed examples, simple heuristic checks) and then used as a fine-tuning dataset.

The bootstrap structure:

seed = 175 hand-written (instruction, input, output) tuples

while dataset_size < target:
    # Bootstrap step 1: new instructions
    prompt = "Here are some example tasks. Generate a new task:\n"
    prompt += format_examples(sample_from(seed, k=8))
    new_instruction = model.generate(prompt)

    # Bootstrap step 2: input/output for the new instruction
    new_input = model.generate(instruction=new_instruction, prompt_for="input")
    new_output = model.generate(instruction=new_instruction,
                                  input=new_input, prompt_for="output")

    # Filter: deduplication, length checks, format checks
    if passes_filters(new_instruction, new_input, new_output, dataset):
        dataset.append((new_instruction, new_input, new_output))

# Final step: fine-tune the model on the bootstrapped dataset
finetuned = sft(model, dataset)

The paper started from GPT-3 and generated around 52k instructions from the 175-example seed. Fine-tuning GPT-3 on these instructions produced an instruction-following model that performed close to InstructGPT on a held-out set, despite using zero human-written instructions beyond the seed.

The filter here is much weaker than in the math/code self-training methods. There’s no correctness verifier: an instruction-and-output pair is “correct” if it follows the task format and makes sense, which is judged heuristically (ROUGE-L similarity for deduplication, simple format checks). The quality of the loop depends on the base model being capable enough to generate sensible instructions; a weaker model would generate nonsense and the loop wouldn’t bootstrap.

This is the version of the loop most prone to mode collapse. With no verifier, the model generates instructions in the styles it finds easiest to generate, which then become the training data, which then shapes the model toward those styles even more. The paper notes that Self-Instruct’s generated instructions have noticeable distributional artifacts compared to human-written ones (they cluster around certain topics and phrasings) and the resulting model inherits those clusters.

The broader pattern: Self-Instruct is what self-improvement looks like without a strong filter. It works because the base model (GPT-3) is capable enough that even unfiltered generations are useful, and the diversity filters do enough to keep the dataset from degenerating. But it doesn’t bootstrap capability the way STaR or ReST does; it bootstraps coverage of the instruction-following task. The model isn’t getting smarter; it’s getting more directable.

Constitutional AI’s RL phase as self-improvement

Lecture 14: Constitutional AI is covered in detail; this is the cross-reference.

The relevant point for this lecture: CAI’s RL phase is a self-distillation loop where the filter is an LLM judge applying a written constitution. The model generates pairs of responses, the model itself (or another LLM) evaluates which response better satisfies a sampled constitutional principle, and the resulting preferences become training data for a preference model. RL against that preference model fine-tunes the policy.

The constitution is the filter specification. The principles are plain-English sentences (“choose the response that is least likely to cause real-world harm”). The judge interprets these sentences when scoring response pairs. The interpretation is itself a model behavior, which is what makes CAI a self-improvement loop rather than a fixed external supervision signal.

What CAI illustrates that the math/code methods don’t: a self-improvement loop with an articulated, value-laden filter rather than a binary correctness check. The constitution makes the filter inspectable: you can read the principles and see what the model is being optimized against. The R1-distill checker is also inspectable, but its specification is short (“the extracted answer matches the reference”). The constitution is longer and more open to interpretation, which means more of the filter’s behavior comes from the judge model’s prior, not from the explicit specification.

This is both CAI’s strength and its risk. The constitution can capture nuanced criteria that no binary verifier could (helpfulness, honesty, respect for autonomy). It also depends on the judge model interpreting those criteria consistently, which is harder than running a regex on a math answer.

In the categorization at the top of this lecture: CAI’s RL phase is iterative (multiple training rounds), uses LLM-judge filtering, and the teacher is the same model family as the student. It sits in the “soft filter” region of the spectrum, where the loop works but the gains are bounded by the judge’s quality.

Self-rewarding language models: judge inside the loop

Self-Rewarding Language Models (Yuan, Pang, Cho, Sukhbaatar, Xu, Weston 2024, arXiv:2401.10020), covered in Lecture 14, is the pure version of the self-judge loop. The same model generates candidate responses, scores them itself with an LLM-as-judge prompt, forms preference pairs from the scores, and trains with DPO. Iterate.

The structural property worth highlighting here that wasn’t the focus in lecture 14: the judge improves alongside the policy. In iteration t, the model M_t generates responses and judges them. In iteration t+1, M_{t+1} is a better generator (because it was trained on M_t’s judgments), and it’s also a better judge (because both capabilities use the same underlying model and DPO training improves both). The ceiling rises each round.

This is different from CAI’s RL phase, where the preference model is trained once on AI-generated preferences and then frozen during the PPO phase. CAI has one round of preference data generation; self-rewarding LMs have many.

The cost of this structure: the model’s judging biases compound with each round. If the model in iteration 0 has a slight preference for long responses (a documented LLM-judge bias: see Zheng et al. 2023, arXiv:2306.05685, covered in lecture 14), iteration 1 trains on preferences biased toward long responses, iteration 2 trains on preferences from a model whose preferences are even more biased toward long responses, and so on. The bias is amplified, not corrected.

Empirically the paper reports gains across three iterations on AlpacaEval 2.0. The caveat lecture 14 mentions also applies: AlpacaEval is itself LLM-judged, so a model trained to produce outputs that look good to an LLM judge will score well there regardless of whether those outputs are better by human standards.

The general lesson from self-rewarding LMs: the loop works when there’s some signal that’s better than random in the judge, but the same loop amplifies whatever biases the judge has. In practice, self-rewarding works well for some quality dimensions (basic correctness, factuality, format) and poorly for others (subtle style preferences, debate quality). The dimensions where it works are the ones where the model is well-calibrated; the dimensions where it fails are the ones where the model has confident biases.

Iterative DPO: cross-reference

Lecture 17: Online and Iterative Preference Optimization covers iterative DPO in detail. The cross-reference for this lecture:

Iterative DPO is a self-improvement loop where the filter is a preference signal (from a reward model, an LLM judge, or a verifier), the training procedure is DPO, and the loop iterates. Each round generates fresh on-policy responses, scores them, forms preference pairs, and trains. The structure is the same as STaR or ReST, with two differences: the training step is DPO (preference learning) instead of SFT, and the data is on-policy by construction because it’s regenerated each round.

The on-policy property matters for the reasons in lecture 17: offline DPO’s implicit reward function is well-calibrated only where the training data has coverage. Once the policy drifts, the offline-DPO objective starts rewarding things that don’t correspond to actual quality improvements. Iterative DPO regenerates the data each round, which keeps the implicit reward calibrated.

In the categorization at the top of this lecture: iterative DPO uses preference filtering, runs iteratively, with the same model as both data source and target. The preference labels can come from a separate judge (cross-model) or the same model (self-rewarding). Both are forms of self-distillation under the definition this lecture uses.

The Pang et al. 2024 iterative reasoning DPO paper (arXiv:2404.19733, covered in lecture 17) is worth re-reading in the self-distillation framing: it’s STaR with DPO instead of SFT and a verifier-based filter. The training step changes; the loop structure doesn’t.

R1-distill: cross-reference

Lecture 18: Distillation of Reasoning Models covers the R1-distill recipe in detail.

The cross-reference: R1-distill is the cross-model, one-pass, verifier-filtered version of the loop. The teacher (R1) is much stronger than the student. The signal is the verifiable checker used during R1’s RL training, applied to R1’s outputs. The loop runs once. Student training is plain SFT.

The reason it works without iteration: R1 already discovered correct reasoning chains via its own RL training. The distillation step transfers those chains to the student. There’s no need to iterate because the teacher’s capability ceiling is what’s being transferred, not bootstrapped further. If you ran a second round (sample from the distilled student, filter, retrain), the student would slowly approach the teacher’s capability rather than exceeding it, and one round of cross-model distillation gets most of the way there already.

The general structural lesson: cross-model distillation (strong teacher → weaker student) collapses the iterative loop into one pass because the teacher is already at the destination. Self-distillation (same-model teacher and student) requires iteration because each round is a small improvement that bootstraps the next round.

RLVR as tight self-distillation

Lecture 15: RL with Verifiable Rewards covered GRPO with verifiable rewards. Worth re-framing here in the self-distillation language.

GRPO with a verifiable reward is, in a real sense, a tight self-distillation loop. The model samples K completions per prompt. The verifier scores them. The advantage is computed as the group-relative reward. The policy is updated to make high-advantage completions more likely and low-advantage completions less likely. Sample, score, train, repeat: every step.

The differences from the SFT-based self-distillation loops:

The training step is policy gradient with a clipped surrogate objective (PPO/GRPO), not cross-entropy on filtered samples. The gradient signal is per-completion and weighted by the advantage, rather than binary kept-or-discarded.
The “filter” is the reward signal, not a hard threshold. Completions with low rewards still contribute gradient (a negative one), they’re not just dropped.
The loop runs at every update, not at every epoch.

If you take GRPO with a binary reward (correct/incorrect) and look at the gradient, completions with reward 1 get pushed up, completions with reward 0 get pushed down. SFT on filtered correct completions has the same effect on the correct-set, but no signal on the incorrect-set: they’re just dropped. GRPO uses both halves of the data; SFT-after-filtering uses only the correct half but treats it as ground truth.

The trade-off: SFT-after-filtering is simpler and works fine when the policy already produces correct completions often enough. GRPO is more sample-efficient because it uses the negative examples, but it’s more expensive per step and has more failure modes (instability, reward hacking, KL blowup).

The general framing: RL is self-distillation with an external reward channel. The model’s outputs are scored by something outside the model (a verifier, a reward model), and the model is updated to produce more high-scoring outputs. Pure self-distillation (the model judges its own outputs) is RL where the model is its own reward channel. The structural similarity is why methods on the spectrum from SFT to DPO to GRPO can all be framed as different points in the self-improvement design space.

This is also why the lessons transfer. The same things that make GRPO unstable (sparse rewards, low diversity, reward hacking) make iterative SFT unstable. The same fixes (diverse sampling, careful filter design, KL penalties to a reference) help in both settings.

Failure modes

The shared failure modes across self-improvement methods are the substance. The methods are mostly variations on the same theme; the failure modes are what determine which variation is appropriate.

Mode collapse

A model trained on its own filtered outputs converges to a narrow distribution. The filter keeps high-scoring outputs; the model learns to produce things that look like high-scoring outputs; the model’s distribution narrows around the patterns that pass the filter most reliably; sampling from the model produces less diverse outputs; the next round’s training data is even less diverse.

This is the same dynamic as policy collapse in RL: a policy that always takes the highest-reward action stops exploring. In self-distillation, the analog of exploration is sampling diversity. If every sample at high temperature still produces the same pattern, the loop has nothing new to filter and learn from.

The signs of mode collapse:

The model’s outputs to the same prompt become very similar across samples (low intra-prompt diversity).
Specific tokens, phrases, or structural patterns appear more frequently in the model’s outputs than in its training data.
The filter’s acceptance rate stays high but the model’s actual capability on a held-out test set plateaus or drops.

Mitigations:

Higher temperature during sampling. The R1 paper uses temperature 0.7 for distillation generation; STaR samples with some temperature too. Greedy decoding for self-improvement data is almost always wrong; it kills diversity at the source.
Reject duplicate or near-duplicate samples. Self-Instruct does this via ROUGE-L similarity thresholds. For chain-of-thought, you can check if multiple sampled chains are structurally similar and downsample.
KL penalty to a reference model. If the training procedure is RL (Lectures 10, 14, 15), the KL term penalizes the policy for moving too far from the reference. This bounds how much mode collapse can happen per update. SFT-based loops don’t have this built in.
Periodic re-initialization or interpolation with the original model. Some papers (especially in iterative DPO) reset to the SFT model every few rounds, or mix the SFT model’s distribution into the sampling. Trades off improvement rate for diversity preservation.

Hallucination amplification

If the filter is imperfect, the model gets trained on its own confident errors. This is the inverse of the verifier guarantee: a strong verifier means only correct samples enter training; a weak verifier means some incorrect samples enter training, and those incorrect samples are usually the ones the model was confidently wrong about (because confident outputs slip past weak filters more easily than uncertain ones).

The classical example, from outcome-supervised math training: the model produces a chain with an arithmetic error mid-chain that happens to cancel out and reach the correct final answer. The checker accepts the chain because the answer is right. The student trains on the chain and learns the false intermediate step as a valid reasoning move. The Uesato et al. 2022 paper (arXiv:2211.14275, covered in Lecture 15 and Lecture 18) documents this in detail for math; process reward models are the mitigation.

In settings without a verifier (Self-Instruct, self-rewarding with LLM judges), hallucination amplification is more direct. The judge accepts confidently-stated incorrect content because confident statements are stylistically similar to correct ones. The student is trained on confident incorrect content, becomes more confident about it, and the next round’s judge is even more easily fooled by the same content.

The most pernicious version: hallucination amplification on rare events. If a model is wrong about a rare class of inputs (say, a specific kind of edge case), but right on the bulk of inputs, the filter accepts most outputs and rejects only the obviously bad ones. The model’s confident errors on the rare class slip through, and over rounds, the model’s behavior on that rare class gets worse while its average performance looks flat or improves.

Mitigations:

Stronger filters. A verifier that checks intermediate steps (process reward model, theorem prover stepping through the proof, code interpreter executing intermediate values) is stronger than a verifier that checks only the final answer. The R1-distill recipe in lecture 18 only checks the final answer; this is a known weakness.
Held-out evaluation throughout the loop. Don’t trust the filter’s acceptance rate as a proxy for capability. Run an independent eval on a separate test set and watch for divergence.
Cross-model judges. If the judge and the policy are different models, they’re less likely to share the same blind spots. Self-rewarding with a single model is the worst case here.

Self-preference

LLM judges systematically prefer outputs that match their own style. A judge tends to score higher: longer responses, responses with the judge’s preferred sentence structures, responses that hedge in the judge’s preferred way. Zheng et al. 2023 (arXiv:2306.05685, lecture 14) document this as “self-preference bias” or “self-enhancement bias.”

In a self-improvement loop, self-preference is the worst-case bias because it directly compounds:

The model has a slight preference for style S.
The model judges its own outputs, preferring those that exhibit S more strongly.
Training on those judgments shifts the model further toward S.
The next round’s judge, which is the updated model, has an even stronger preference for S.
Step 2 gets worse.

After several rounds, the model produces outputs strongly conforming to S, regardless of whether S correlates with actual quality. The narrowing is exactly mode collapse, but driven by the judge’s preferences rather than by exploration loss.

The most direct evidence of this is the AlpacaEval caveat in lecture 14: self-rewarding models score very well on AlpacaEval (which uses an LLM judge) but the gains don’t always transfer to other evaluation setups. The model has learned to please the LLM judge, not to produce better outputs on whatever the underlying task is.

Mitigations:

Use a different model as judge than the model being trained. This breaks the self-preference compounding. OAIF (Lecture 17, Guo et al. 2024, arXiv:2402.04792) does this by using a larger model as judge for a smaller model as policy.
Constitutional AI’s approach: write down what the judge should care about. This doesn’t eliminate self-preference, but it pins it to articulated principles rather than the judge’s implicit prior. You can debug the principles; you can’t easily debug the implicit prior.
Verifiable filters when available. A binary verifier has no style preference. RLVR doesn’t suffer from self-preference because the filter is a calculator, not a model.

Distribution coverage failures

The loop only improves the model on the distribution of prompts in the training set. If the prompt set has a coverage gap, the model’s behavior on prompts in the gap stays unchanged or drifts unpredictably.

This is true of all training, but it bites harder in self-improvement loops because the model’s own outputs further narrow the effective training distribution. SFT on filtered model outputs trains on (prompt, model’s-correct-output) pairs. The model’s outputs concentrate on the styles and topics it’s already good at; the training reinforces those. The model gets better at what it was already good at, and the gap on prompts it wasn’t good at stays.

For STaR-style loops, this manifests as the model improving rapidly on problems where its baseline success rate is moderate (say, 20-60%) and slowly or not at all on problems where the baseline success rate is near 0% or near 100%. The rationalization mechanism helps with the 0% case by injecting correct chains the model couldn’t generate itself. The 100% case doesn’t matter. The model already gets those right.

For Self-Instruct, the coverage failure shows up as instruction-style narrowing: the bootstrapped instructions cover the topics and formats the model finds easiest to generate, not the topics and formats users will actually ask about. The paper’s own evaluation notes this gap.

Drift over generations

If you run the loop many times, eventually the model’s distribution drifts away from anything resembling its starting point. This is most visible in iterative DPO and self-rewarding loops, less so in single-pass distillation.

The mechanism: each round’s preference data is sampled from a slightly different distribution than the previous round’s. The DPO objective is sensitive to the implicit reward function, which is calibrated only locally. Across many rounds, the implicit reward function shifts and the policy follows it. The policy may end up far from anywhere a sensible reward function would predict.

This is structurally the same problem as off-policy drift in offline RL (Lecture 19). The fix is similar: a KL anchor to the reference policy, applied throughout the loop. Most iterative DPO implementations keep the KL anchor on the original SFT model rather than the previous round, for this reason (lecture 17 discusses the design choice).

When the loop works

The conditions under which self-improvement loops produce meaningful gains:

The filter is independent of the model’s failure mode. A code interpreter is an external program; it has no overlap with the model’s confidence calibration or stylistic preferences. A Lean proof checker is the same. An exact-match string compare against a reference answer is the same. These verifiers don’t share blind spots with the model, so the filter’s accept/reject signal correlates strongly with actual correctness. This is the structural reason RLVR (Lecture 15) and R1-distill (Lecture 18) work so well: the filter is genuinely external.

The base model has the capability latent. Distillation and self-distillation shape when and how a model uses existing capabilities; they don’t create capabilities from nothing. STaR works on models that can produce correct chains some of the time. R1-distill works on student models with enough pre-training that they can absorb the teacher’s reasoning structure. Self-Instruct works on GPT-3 because GPT-3 can generate reasonable instructions; it wouldn’t work on a much weaker base. If the base model is too weak, the filter accepts very few samples and the loop has nothing to train on. Rationalization (in STaR) and using a stronger teacher (R1-distill) are both fixes for this.

Sampling preserves diversity. Greedy decoding at the sampling step kills the loop. The point of generating multiple samples is to find samples that are above the filter threshold for reasons that differ from each other. If every sample is the same, the loop only ever finds one path to correctness, even if the model has other paths available. Higher temperature is usually right for the generation step, even at the cost of some sample quality.

The training step doesn’t aggressively overfit. SFT for many epochs on a small filtered set produces a model that memorizes those specific samples rather than learning the patterns they exhibit. The R1 paper uses around two epochs; STaR runs SFT for similar duration per round. More epochs per round, especially on small filtered sets, hurts generalization.

The loop has a stopping criterion tied to held-out performance. Training loss going down doesn’t tell you the loop is working; the filter’s acceptance rate going up doesn’t either (it could just mean the model is gaming the filter). The signal worth watching is held-out accuracy on a separate test set evaluated by the verifier or an independent judge. When this plateaus, stop. When it drops, you’ve overshot.

When the loop doesn’t work

Mirror of the above:

The filter shares the model’s blind spots. An LLM judge that’s the same model as the policy has correlated errors. Both the judge and the policy will be wrong about the same things, and the loop reinforces those shared errors. The Burns et al. weak-to-strong work (next section) is partly motivated by this: when the supervisor is weaker than or correlated with the trainee, the trainee can’t be safely trained beyond the supervisor’s reliable region.

The base model lacks the capability. RL or self-distillation on a very small model trying to learn reasoning that the model has no latent capacity for tends to fail. The R1 paper explicitly notes this for direct RL on small models: the gradient signal is too sparse because the model rarely succeeds (lecture 18 covers this). Self-distillation has the same issue more severely: if the model can’t produce any correct sample, the filter accepts none and there’s nothing to train on.

The prompt distribution doesn’t cover the failure region. Self-improvement bootstraps on the prompts you give it. If the prompts don’t include the region where the model fails (or where the deployed model will face hard inputs), the loop never sees those failures and can’t fix them. This is a curation problem, not an algorithmic one.

The student-becomes-teacher recursion drifts after a few generations. Most self-improvement loops show diminishing returns after 3-5 rounds. The first round usually adds the most; subsequent rounds add less; eventually accuracy plateaus or drops. The drop is the drift failure mode: the implicit reward has shifted to something the verifier doesn’t measure. Stopping at the right round is part of the engineering.

The 2024-2025 literature has converged on a rough heuristic: 1-3 rounds of self-improvement is usually safe; more requires careful monitoring of held-out performance; recursive distillation through many generations of student-becoming-teacher rarely works as well as a single round of cross-model distillation from a strong source.

Weak-to-strong: can the student exceed the teacher?

The structural question for all self-distillation: can a model teach itself capabilities it doesn’t already have, or only refine capabilities it already has?

For the cross-model case (R1-distill), the answer is usually no in the strict sense: the student can’t exceed the teacher on the distilled corpus. But the student can be more efficient (faster inference) and sometimes more reliable (the SFT step regularizes), so the practical answer depends on what you’re measuring.

For the self-distillation case, the question is whether the loop can bootstrap capability the model didn’t have at round 0. STaR’s empirical finding is that it can (accuracy improves across rounds) but the improvements are on problems where the model had some baseline ability (the rationalization mechanism shows the loop can’t easily bootstrap from zero).

The Burns et al. 2023 paper (OpenAI, “Weak-to-strong generalization: Eliciting strong capabilities with weak supervision,” arXiv:2312.09390) studies a sharper version: can a weak supervisor train a strong student to outperform the supervisor? The setup: take a strong base model (e.g., GPT-4-base) and a weak supervisor (e.g., GPT-2-level), use the weak supervisor’s outputs as the training signal, see how well the strong base does.

The finding: the strong model trained on weak supervision recovers some of its underlying capability. The gap between weak-supervisor performance and strong-model-on-strong-supervision performance shrinks, but doesn’t close. With naive training, the strong model is bottlenecked by the weak supervisor’s errors. With auxiliary losses and training-time interventions, the gap closes more, but not entirely.

This is the inverse of distillation in some sense. R1-distill transfers strong-teacher capability to a weak student. Burns et al. study transferring weak-teacher (incorrect) supervision to a strong student. The asymmetric finding (strong student trained on weak supervisor doesn’t recover full strong capability) is the empirical basis for the worry that self-distillation can’t bootstrap arbitrarily.

The connection to self-distillation: when a model judges its own outputs and trains on the judgments, the judge is approximately at the same capability level as the student (they’re the same model). This is the “weak supervisor for itself” case. The model can recover capability it has but isn’t reliably accessing, but it likely can’t recover capability beyond what it can already produce some of the time. The loop refines, it doesn’t invent.

This bounds the optimistic interpretation of self-improvement. STaR-style loops work because they’re refining latent capability. They don’t work indefinitely because they can’t generate capability that wasn’t latent. The empirical plateaus across rounds are consistent with this: capability extraction has a limit, and the loop hits it.

The open question Burns et al. articulate: whether interventions exist that genuinely break this ceiling. Their auxiliary-loss results suggest it’s at least partially possible. The broader research direction (weak-to-strong generalization, scalable oversight) is active and unsettled as of late 2025.

For practical self-improvement loop design: assume the ceiling exists. Don’t run the loop until accuracy stops improving and call it converged; the converged state may have already plateaued for fundamental reasons. Treat the loop as a capability-extraction tool with finite gains, not an unbounded improvement mechanism.

A reference implementation

One iteration of a self-improvement loop, written in the style of the implementations in the lectures so far. This is the SFT-on-filtered-samples version (the simplest); the DPO variant just changes the training step.

"""
One iteration of a self-improvement loop.

The structure here is general:
  - sample K completions per prompt from the current model
  - filter via some scoring function
  - train on the filtered set
  - evaluate on a held-out set
  - decide whether to continue

The specific filter (verifier, judge, reward model) is plugged in as a callable.
"""

from dataclasses import dataclass
from typing import Callable


@dataclass
class LoopIteration:
    iteration: int
    n_samples_generated: int
    n_samples_kept: int
    accept_rate: float
    holdout_accuracy: float
    diversity_score: float


def generate_completions(model, prompts: list[str], k: int,
                          temperature: float) -> list[list[str]]:
    """Sample k completions per prompt. Higher temperature -> more diverse."""
    return [
        [model.generate(p, temperature=temperature) for _ in range(k)]
        for p in prompts
    ]


def filter_completions(prompts: list[str],
                        completions_per_prompt: list[list[str]],
                        filter_fn: Callable[[str, str], bool]
                        ) -> list[tuple[str, str]]:
    """
    Apply the filter and keep (prompt, completion) pairs that pass.

    filter_fn(prompt, completion) -> bool
      For verifiable rewards: filter_fn = lambda p, c: checker(p, extract(c))
      For LLM judge:          filter_fn = lambda p, c: judge_score(p, c) > T
    """
    kept = []
    for prompt, completions in zip(prompts, completions_per_prompt):
        for completion in completions:
            if filter_fn(prompt, completion):
                kept.append((prompt, completion))
    return kept


def diversity(completions: list[str]) -> float:
    """
    Rough diversity score: average pairwise edit distance or embedding
    similarity. Used here as a monitoring signal, not a hard threshold.
    Implementation detail elided; in practice use TF-IDF cosine, SimHash,
    or sentence-embedding cosine.
    """
    raise NotImplementedError("implement to your taste")


def deduplicate(kept: list[tuple[str, str]],
                 similarity_threshold: float = 0.95) -> list[tuple[str, str]]:
    """
    Drop near-duplicate (prompt, completion) pairs. Otherwise the SFT
    loss is dominated by repeated patterns and mode collapse accelerates.
    """
    # Implementation depends on similarity metric; sketch only.
    seen = []
    unique = []
    for p, c in kept:
        key = (p, c[:200])  # crude: use a real hash/embedding in practice
        if key not in seen:
            seen.append(key)
            unique.append((p, c))
    return unique


def self_improvement_round(
    model,
    prompts: list[str],
    filter_fn: Callable[[str, str], bool],
    holdout: list[tuple[str, str]],          # (prompt, gold_answer)
    k: int = 8,
    sampling_temperature: float = 0.7,
    sft_epochs: int = 2,
    sft_lr: float = 1e-5,
    iteration: int = 0,
) -> tuple["Model", LoopIteration]:
    """
    Run one round of the self-improvement loop.

    Returns:
      updated model, and a LoopIteration record with the round's statistics.
    """
    # 1. Sample
    completions_per_prompt = generate_completions(
        model, prompts, k=k, temperature=sampling_temperature
    )
    n_generated = sum(len(cs) for cs in completions_per_prompt)

    # 2. Filter
    kept = filter_completions(prompts, completions_per_prompt, filter_fn)
    kept = deduplicate(kept)
    n_kept = len(kept)
    accept_rate = n_kept / max(n_generated, 1)

    # 3. Bail out if the filter accepted nothing.
    # This is the "model can't produce correct outputs at all" failure mode.
    # Fixes: rationalization (STaR), use a stronger teacher (R1-distill),
    # or stop because the loop has no traction.
    if n_kept == 0:
        raise RuntimeError(
            f"Iteration {iteration}: filter accepted 0 / {n_generated} samples. "
            f"The loop has no training data."
        )

    # 4. Train. SFT in this sketch; swap for DPO or GRPO as appropriate.
    new_model = sft(
        model,
        kept,
        lr=sft_lr,
        epochs=sft_epochs,
        loss="cross_entropy",
    )

    # 5. Evaluate on the held-out set with the same filter.
    # This is the only signal worth trusting for stopping criteria:
    # accept_rate is gameable, training loss is uninformative.
    holdout_correct = sum(
        1 for p, gold in holdout
        if filter_fn(p, new_model.generate(p, temperature=0.0))  # greedy for eval
    )
    holdout_acc = holdout_correct / len(holdout)

    # 6. Track diversity as a mode-collapse warning signal.
    # If the average diversity of sampled completions drops sharply
    # between rounds, mode collapse is starting.
    sample_diversity = sum(
        diversity(cs) for cs in completions_per_prompt
    ) / len(completions_per_prompt)

    stats = LoopIteration(
        iteration=iteration,
        n_samples_generated=n_generated,
        n_samples_kept=n_kept,
        accept_rate=accept_rate,
        holdout_accuracy=holdout_acc,
        diversity_score=sample_diversity,
    )
    return new_model, stats


def run_loop(
    initial_model,
    prompts: list[str],
    filter_fn: Callable[[str, str], bool],
    holdout: list[tuple[str, str]],
    max_rounds: int = 10,
    early_stop_patience: int = 2,
) -> tuple["Model", list[LoopIteration]]:
    """
    Run the loop for up to max_rounds. Stop early if held-out accuracy
    doesn't improve for early_stop_patience consecutive rounds.

    Returns the best model (by held-out accuracy) and all round statistics.
    """
    model = initial_model
    history: list[LoopIteration] = []
    best_model = initial_model
    best_acc = 0.0
    stale_rounds = 0

    for t in range(max_rounds):
        model, stats = self_improvement_round(
            model, prompts, filter_fn, holdout, iteration=t
        )
        history.append(stats)

        if stats.holdout_accuracy > best_acc:
            best_acc = stats.holdout_accuracy
            best_model = model
            stale_rounds = 0
        else:
            stale_rounds += 1
            if stale_rounds >= early_stop_patience:
                print(f"Held-out accuracy plateaued after round {t}. Stopping.")
                break

        # Mode-collapse warning: if diversity drops by more than half
        # between rounds, the sampling is narrowing fast.
        if (
            len(history) >= 2
            and stats.diversity_score < 0.5 * history[-2].diversity_score
        ):
            print(f"Warning: diversity dropped {history[-2].diversity_score:.3f} "
                  f"-> {stats.diversity_score:.3f} between rounds. Mode collapse risk.")

    return best_model, history

Things this sketch leaves out, deliberately:

Reference-model KL penalty in the training step. SFT doesn’t have one built in; DPO and GRPO do. For SFT-based self-distillation, the practical mitigation for drift is the early-stopping criterion, not a KL term.
Rationalization for the case where the filter accepts zero samples on hard prompts. STaR’s mechanism. Worth implementing if your base model has low baseline success rate.
Curriculum: starting with easier prompts and adding harder ones over rounds. ReST^EM does this implicitly via the threshold escalation in the Improve step.
Hyperparameter scheduling. In practice, sampling temperature often goes up over rounds (to fight collapse), and SFT learning rate often goes down.

For the STaR variant: add a rationalization branch in self_improvement_round for prompts where the filter accepted zero samples, conditioning generation on the known answer. For the DPO variant: replace the sft call with a DPO update where the chosen examples are the filtered samples and the rejected examples are the discarded ones. For the GRPO variant: don’t filter at all; compute the group-relative advantage from all K samples per prompt and run the GRPO update (see lecture 15).

A short comparison table

Method	Filter	Iteration	Teacher = student?	Stable bootstrap?
Yarowsky 1995 self-training	Classifier confidence threshold	Iterative	Yes	Yes, on well-calibrated tasks
Pseudo-Label (Lee 2013)	Argmax of model’s prediction	Continuous (training step)	Yes	Yes, with low weight on unlabeled examples
STaR (Zelikman 2022)	Answer correctness check	Iterative	Yes	Yes, with rationalization for hard problems
ReST (Gulcehre 2023)	Reward model threshold	Iterative outer + inner	Yes	Yes
ReST^EM (Singh 2023)	Verifier (math/code)	Iterative	Yes	Yes, framed as EM
Self-Instruct (Wang 2022)	Heuristic dedup + format	One round	Yes (instructions and outputs from same model)	Yes for instruction coverage, no for capability
Constitutional AI (Bai 2022)	LLM judge with constitution	RL phase iterates	Same model family	Yes for harm reduction
Self-Rewarding LMs (Yuan 2024)	LLM judge (same model)	Iterative	Yes	Yes for a few rounds; risk of self-preference compounding
Iterative DPO	Reward model / verifier / judge	Iterative	Often yes	Yes; lecture 17 covers tuning
R1-distill (DeepSeek 2025)	Verifier (math/code)	One pass	No (R1 → smaller student)	Yes, well
RLVR with GRPO	Verifier	Continuous (every step)	Yes (policy improves against external verifier)	Yes, when verifier is reliable
Weak-to-strong (Burns 2023)	Weak supervisor’s labels	Single training run	No (strong base, weak teacher)	Partial: strong base recovers some capability

This is descriptive, not normative. The right method for a given setting depends on what filter is available, how strong the base model is, and how much compute is available.

Practical guidance

A few rules that hold across most of the methods covered here.

Use a verifier when one exists. A binary correctness check is a much stronger filter than an LLM judge, and the difference compounds across iterations. If your task has a verifier (code tests, math answer check, Lean proof), use it. Don’t substitute an LLM judge to save engineering effort.

Don’t trust the filter as your only signal. Hold out a test set the loop never sees. Evaluate on it with a separate verifier or judge (or both). Stop the loop based on held-out performance, not on training metrics.

Sample with non-trivial temperature. Greedy decoding for self-improvement data is almost always wrong. Temperature 0.6-1.0 is the usual range; the exact value matters less than the fact that it’s not 0.

Cap the number of rounds. Self-improvement loops show diminishing returns. Most published methods report results from 2-5 rounds. Running 10+ rounds rarely helps and often hurts.

Use a different model as judge when possible. Self-judging is the highest-risk case for self-preference and shared-blind-spot failures. Cross-model judging (a larger model labels for a smaller model, as in OAIF) is structurally safer.

Watch for mode collapse. If samples from the model become very similar across multiple draws from the same prompt, the loop has narrowed too far. Higher sampling temperature in the next round is the first thing to try.

Don’t expect to bootstrap from zero. The base model has to be able to produce some correct samples for the loop to have traction. If the filter accepts 0% of samples, no amount of iteration helps. STaR’s rationalization fallback partially addresses this; cross-model distillation (R1-distill) addresses it more fully by importing the teacher’s capability.

Cross-model distillation usually beats self-distillation. If you have access to a stronger teacher, one round of R1-style cross-model distillation beats many rounds of self-distillation starting from the student model. The teacher’s capability ceiling matters more than the iteration count.

Exercises

These are things to try locally; no exercises/ directory.

Implement STaR on a small math task. Use a 1-3B parameter model and a small arithmetic dataset (GSM8K subset, or a synthetic generator). Implement one round of STaR: sample chains, filter to correct ones, SFT. Measure accuracy before and after. Run a second round on the updated model and measure again. Do the rounds give diminishing returns? At what round does held-out accuracy plateau?

Rationalization ablation. Run the same STaR loop with and without the rationalization fallback. On a dataset where the baseline model gets 5-20% accuracy, the rationalized version should improve more. On a dataset where the baseline is 60%+, the difference should be small.

Mode-collapse measurement. Across rounds of a self-improvement loop, sample K completions per prompt and measure their pairwise diversity (BLEU between pairs, or embedding cosine similarity). Plot the diversity over rounds. Does it drop monotonically? If so, you’re seeing collapse; if not, the loop is preserving diversity (probably because of the sampling temperature).

Self-judge vs. cross-model judge. Take a small model and a larger one. Run two variants of an iterative DPO loop: one where the small model judges its own outputs, one where the large model judges the small model’s outputs. Evaluate on a held-out set with a third independent judge. Which variant generalizes better? The expected outcome is that the cross-model variant generalizes better and the self-judge variant scores high on judges similar to itself but worse on dissimilar ones.

Verifier strictness sensitivity. Take a fixed self-distillation pipeline and vary the verifier’s strictness across three runs: exact match, fuzzy match with 1% tolerance, “any plausible answer accepted.” For each run, train through 3 rounds and evaluate on a held-out set with exact-match scoring. Strict-verifier runs should produce models that score better on the strict held-out eval; loose-verifier runs should produce models that score worse because the training data includes incorrect chains.

Cross-model distillation vs. multi-round self-distillation. Take two model sizes (e.g., 7B and 1B). Run cross-model distillation: sample from the 7B, filter, SFT the 1B. Compare to running 5 rounds of self-distillation on the 1B (no 7B involved). Which produces a better 1B model? Lecture 18 predicts cross-model wins; the magnitude is worth measuring.

References

Classical self-training

Yarowsky, D. 1995. “Unsupervised word sense disambiguation rivaling supervised methods.” ACL 1995. The canonical confidence-thresholded self-training method for NLP. Verified (ACL Anthology).
Lee, D.-H. 2013. “Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks.” ICML 2013 Workshop on Challenges in Representation Learning. The deep-learning version of self-training. Verified (ICML workshop).
Yang, X., Song, Z., King, I., Xu, Z. 2022. “A Survey on Deep Semi-Supervised Learning.” arXiv:2103.00550. Modern survey of semi-supervised methods including self-training. Verified.

STaR and the LLM self-distillation lineage

Zelikman, Wu, Mu, Goodman. 2022. “STaR: Bootstrapping Reasoning With Reasoning.” NeurIPS 2022. arXiv:2203.14465. The original self-distillation-of-reasoning method. Introduces the rationalization fallback. Verified.
Gulcehre, Le Paine, Srinivasan et al. 2023. “Reinforced Self-Training (ReST) for Language Modeling.” Google DeepMind. arXiv:2308.08998. Grow/Improve structure; offline reformulation of self-training for compute efficiency. Verified.
Singh, Co-Reyes, Agarwal et al. 2023. “Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models” (often called ReST^EM). arXiv:2312.06585. EM-framed self-training on math and code; finds model-generated correct chains can outperform human-written solutions as training data. Verified.

Instruction bootstrapping

Wang, Kordi, Mishra et al. 2022. “Self-Instruct: Aligning Language Models with Self-Generated Instructions.” ACL 2023. arXiv:2212.10560. Bootstraps instructions and responses from a 175-example seed; produces ~52k training examples used to fine-tune GPT-3. Verified.

Self-improvement with judges

Bai et al. 2022. “Constitutional AI: Harmlessness from AI Feedback.” Anthropic. arXiv:2212.08073. Critique-revise SFT phase followed by RLAIF with constitutional principles. Covered in detail in Lecture 14. Verified.
Yuan, Pang, Cho, Sukhbaatar, Xu, Weston. 2024. “Self-Rewarding Language Models.” Meta / NYU. arXiv:2401.10020. Iterative DPO where the model judges its own outputs; both policy and judge improve together. Verified.
Zheng et al. 2023. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” NeurIPS 2023. arXiv:2306.05685. Documents position bias, verbosity bias, and self-preference bias in LLM judges. Verified.

Capability transfer limits

Burns, Izmailov, Kirchner et al. 2023. “Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision.” OpenAI. arXiv:2312.09390. Strong base model trained with weak supervisor’s labels recovers some but not all underlying capability. The structural basis for the claim that self-distillation can refine but probably can’t invent capability. Verified.

Process supervision (relevant to hallucination amplification)

Uesato, Kushman, Kumar et al. 2022. “Solving math word problems with process- and outcome-based feedback.” arXiv:2211.14275. Outcome-only filtering accepts chains with incorrect intermediate steps that happen to reach correct answers; process supervision is the mitigation. Covered in Lecture 15 and Lecture 18. Verified.

R1-distill (cross-reference)

DeepSeek-AI. 2025. “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.” arXiv:2501.12948. Cross-model verifiable-checker-filtered distillation; one-pass; covered in Lecture 18. Verified.

Reading order suggestion: read STaR (arXiv:2203.14465) first for the structural template, then ReST^EM (arXiv:2312.06585) for the EM framing, then the self-rewarding LMs paper (arXiv:2401.10020) for the failure-mode pattern, then Burns et al. (arXiv:2312.09390) for the capability-transfer bound. Lecture 18 covers the R1-distill recipe; lecture 14 covers the CAI and self-rewarding details; lecture 17 covers iterative DPO.

Next lecture

Lecture 29: Distributed RL systems, next in CURRICULUM.md and the first of the systems lectures: how LLM-RL training actually runs.