unreviewed draftNo one has checked this end to end. Treat the math, code, and citations as unverified. Source on GitHub.
Reinforcement Learning from Human Feedback (RLHF) is a training paradigm used to align language model outputs with human preferences. The standard pipeline has three stages: supervised fine-tuning (SFT) on high-quality demonstrations, reward model training from pairwise human comparisons, and RL optimization (typically PPO) against that reward model.
The PAPERS.md here is generated by ../../../tools/arxiv-collector/; it lists papers returned by arXiv queries on RLHF, alignment, and preference optimization, filtered to 2022 onward.
Core methods
PPO-based RLHF: the original approach from InstructGPT (Ouyang et al., 2022, arXiv:2203.02155). Requires a separate reward model and carries significant engineering complexity (four model copies in memory simultaneously).
DPO: Direct Preference Optimization (Rafailov et al., 2023, arXiv:2305.18290) eliminates the explicit reward model by reparameterizing the RLHF objective. Simpler to implement; requires preference-paired data.
GRPO: Group Relative Policy Optimization, introduced in DeepSeekMath (Shao et al., 2024, arXiv:2402.03300). Samples multiple completions per prompt and computes advantages from their relative quality, avoiding a separate critic network. Well-suited to tasks with verifiable rewards (test pass rate, math verification).
Constitutional AI: Bai et al., 2022 (arXiv:2212.08073): uses a set of written principles to guide both a supervised critique-revision loop and an RL stage, reducing reliance on human preference labels for harmlessness.
Key challenges
- Reward hacking: models can exploit gaps in the reward model: passing tests with incorrect logic, optimizing superficial style signals. KL penalties to the SFT policy and diverse test suites help limit this.
- Data quality: human preference labels for code require domain expertise and produce noisy, context-dependent judgments.
- Multi-objective tension: correctness, readability, security, and efficiency can pull in different directions; reward models typically approximate some weighted combination.
Practical tooling
- HuggingFace TRL: PPO, DPO, and GRPO trainers for transformers.
- ReaLHF: parameter reallocation system for efficient RLHF training across multi-GPU clusters.
- Standard benchmarks for alignment quality: MT-Bench, AlpacaEval, domain-specific human evaluation.