unreviewed draftNo one has checked this end to end. Treat the math, code, and citations as unverified. Source on GitHub.
Program synthesis here means generating code that satisfies a specification, typically verified by running unit tests. This directory focuses on work where RL is the mechanism for improving synthesis quality beyond what supervised training achieves.
The PAPERS.md here is generated by ../../../tools/arxiv-collector/; it lists papers returned by arXiv queries on program synthesis and RL, filtered to 2022 onward.
Landmark papers
AlphaCode: Li et al., DeepMind, Science 2022. arXiv:2203.07814. Achieved top-54% performance on Codeforces by generating large numbers of candidate programs and filtering/clustering to a final submission set. Uses an offline RL algorithm (GOLD) rather than online PPO.
CodeRL: Le et al., Salesforce Research, NeurIPS 2022. arXiv:2207.01780. Treats the code-generating LM as an actor and trains a critic network to predict functional correctness, enabling dense RL feedback during training.
DeepSeek-R1: DeepSeek-AI, 2025. arXiv:2501.12948. Applies GRPO with verifiable rewards at scale; strong results on both math reasoning and code.
For a survey of RL applied to code LLMs, see arXiv:2412.20367 (Dec 2024). For a broader LLM-for-code survey (including benchmarks and fine-tuning techniques), see arXiv:2503.01245 (March 2025, also published in ACM TOSEM).
Core challenges
Sparse rewards (a program either passes the test suite or it doesn’t) make RL training harder than in domains with continuous reward signals. CodeRL addresses this with a critic that provides intermediate signal. An alternative is process-level supervision, where feedback is given on intermediate reasoning steps rather than only the final output.
Sandbox isolation is required: code execution during RL training must run in resource-bounded containers to prevent destructive operations.
Recent directions (2024–2025)
Test-time compute scaling and reasoning chains (o1-style, DeepSeek-R1) have improved performance substantially on competition-level benchmarks. These systems use RL to train models to produce extended chain-of-thought before emitting code. For surveys covering these developments, search arXiv for recent work on “reinforcement learning with verifiable rewards” and “test-time compute scaling.”