All collected papers: LLM Code Generation | Study Reinforcement Learning

Collected automatically from arXiv by the repo's paper collector. A search index, not a curated list.

Last Updated: 2025-11-12

Total Papers: 271

2025

Revisiting Entropy in Reinforcement Learning for Large Reasoning Models

Authors: Renren Jin, Pengzhi Gao, Yuqi Ren et al. Published: 2025-11-08 arXiv: 2511.05993v1 PDF: Download

Abstract: Reinforcement learning with verifiable rewards (RLVR) has emerged as a predominant approach for enhancing the reasoning capabilities of large language models (LLMs). However, the entropy of LLMs usually collapses during RLVR training, causing premature convergence to suboptimal local minima and hind…

You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models

Authors: Shuvendu Roy, Hossein Hajimirsadeghi, Mengyao Zhai et al. Published: 2025-11-07 arXiv: 2511.04902v1 PDF: Download

Abstract: Recent advances in large language models have demonstrated the promise of unsupervised reinforcement learning (RL) methods for enhancing reasoning capabilities without external supervision. However, the generalizability of these label-free RL approaches to smaller base models with limited reasoning …

Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models

Authors: Chenxi Liu, Junjie Liang, Yuqi Jia et al. Published: 2025-11-06 arXiv: 2511.04800v1 PDF: Download

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs). The Group Relative Policy Optimization (GRPO) family has demonstrated strong performance in training LLMs with RLVR. However, as models tr…

Reasoning Models Sometimes Output Illegible Chains of Thought

Authors: Arun Jose Published: 2025-10-31 arXiv: 2510.27338v1 PDF: Download

Abstract: Language models trained via outcome-based reinforcement learning (RL) to reason using chain-of-thought (CoT) have shown remarkable performance. Monitoring such a model’s CoT may allow us to understand its intentions and detect potential malicious behavior. However, to be effective, this requires tha…

Greedy Sampling Is Provably Efficient for RLHF

Authors: Di Wu, Chengshuai Shi, Jing Yang et al. Published: 2025-10-28 arXiv: 2510.24700v1 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique for post-training large language models. Despite its empirical success, the theoretical understanding of RLHF is still limited, as learning the KL-regularized target with only preference feedback poses additional challe…

Latent Chain-of-Thought for Visual Reasoning

Authors: Guohao Sun, Hang Hua, Jian Wang et al. Published: 2025-10-27 arXiv: 2510.23925v2 PDF: Download

Abstract: Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs). However, existing training algorithms such as SFT, PPO, and GRPO may not generalize well across unseen reasoning tasks and heavily rely on a biased reward model. T…

BLIP-FusePPO: A Vision-Language Deep Reinforcement Learning Framework for Lane Keeping in Autonomous Vehicles

Authors: Seyed Ahmad Hosseini Miangoleh, Amin Jalal Aghdasian, Farzaneh Abdollahi Published: 2025-10-25 arXiv: 2510.22370v1 PDF: Download

Abstract: In this paper, we propose Bootstrapped Language-Image Pretraining-driven Fused State Representation in Proximal Policy Optimization (BLIP-FusePPO), a novel multimodal reinforcement learning (RL) framework for autonomous lane-keeping (LK), in which semantic embeddings generated by a vision-language m…

LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts

Authors: Siyuan Wang, Gaokai Zhang, Li Lyna Zhang et al. Published: 2025-10-22 arXiv: 2510.19363v2 PDF: Download

Abstract: Reasoning over long contexts is essential for large language models. While reinforcement learning (RL) enhances short-context reasoning by inducing “Aha” moments in chain-of-thought, the advanced thinking patterns required for long-context reasoning remain largely unexplored, and high-difficulty RL …

Enhancing Reasoning Skills in Small Persian Medical Language Models Can Outperform Large-Scale Data Training

Authors: Mehrdad Ghassabi, Sadra Hakim, Hamidreza Baradaran Kashani et al. Published: 2025-10-22 arXiv: 2510.20059v2 PDF: Download

Abstract: Enhancing reasoning capabilities in small language models is critical for specialized applications such as medical question answering, particularly in underrepresented languages like Persian. In this study, we employ Reinforcement Learning with AI Feedback (RLAIF) and Direct preference optimization …

CosmoCore Affective Dream-Replay Reinforcement Learning for Code Generation

Authors: Santhosh Kumar Ravindran Published: 2025-10-20 arXiv: 2510.18895v1 PDF: Download

Abstract: We introduce CosmoCore, a neuroscience-inspired reinforcement learning (RL) architecture that integrates affective signals to enhance code generation in large language models (LLMs). Motivated by human and animal learning where embarrassment from mistakes drives rapid correction, as observed in trai…

UniRL-Zero: Reinforcement Learning on Unified Models with Joint Language Model and Diffusion Model Experts

Authors: Fu-Yun Wang, Han Zhang, Michael Gharbi et al. Published: 2025-10-20 arXiv: 2510.17937v1 PDF: Download

Abstract: We present UniRL-Zero, a unified reinforcement learning (RL) framework that boosts, multimodal language model understanding and reasoning, diffusion model multimedia generation, and their beneficial interaction capabilities within a unified model. Our work defines six scenarios for unified model rei…

The Ends Justify the Thoughts: RL-Induced Motivated Reasoning in LLMs

Authors: Nikolaus Howe, Micah Carroll Published: 2025-10-20 arXiv: 2510.17057v1 PDF: Download

Abstract: The use of reinforcement learning (RL) with chain-of-thought (CoT) reasoning has emerged as a promising approach for developing more capable language models. In turn, this has led to investigation of CoT monitoring as a compelling method for detecting harmful behaviors such as reward hacking, under …

Dual-Weighted Reinforcement Learning for Generative Preference Modeling

Authors: Shengyu Feng, Yun He, Shuang Ma et al. Published: 2025-10-17 arXiv: 2510.15242v2 PDF: Download

Abstract: Reinforcement learning (RL) has recently proven effective at scaling chain-of-thought (CoT) reasoning in large language models on tasks with verifiable answers. However, extending RL to more general non-verifiable tasks, typically in the format of human preference pairs, remains both challenging and…

Unilaw-R1: A Large Language Model for Legal Reasoning with Reinforcement Learning and Iterative Inference

Authors: Hua Cai, Shuang Zhao, Liang Zhang et al. Published: 2025-10-11 arXiv: 2510.10072v1 PDF: Download

Abstract: Reasoning-focused large language models (LLMs) are rapidly evolving across various domains, yet their capabilities in handling complex legal problems remains underexplored. In this paper, we introduce Unilaw-R1, a large language model tailored for legal reasoning. With a lightweight 7-billion parame…

h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning

Authors: Sumeet Ramesh Motwani, Alesia Ivanova, Ziyang Cai et al. Published: 2025-10-08 arXiv: 2510.07312v2 PDF: Download

Abstract: Large language models excel at short-horizon reasoning tasks, but performance drops as reasoning horizon lengths increase. Existing approaches to combat this rely on inference-time scaffolding or costly step-level supervision, neither of which scales easily. In this work, we introduce a scalable met…

Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners

Authors: Xiangchi Yuan, Xiang Chen, Tong Yu et al. Published: 2025-10-06 arXiv: 2510.04454v1 PDF: Download

Abstract: Large Language Models (LLMs) show strong reasoning abilities, often amplified by Chain-of-Thought (CoT) prompting and reinforcement learning (RL). Although RL algorithms can substantially improve reasoning, they struggle to expand reasoning boundaries because they learn from their own reasoning traj…

Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment

Authors: Yunfan Zhang, Kathleen McKeown, Smaranda Muresan Published: 2025-10-05 arXiv: 2510.04045v1 PDF: Download

Abstract: Large Language Models (LLMs) are typically trained to reflect a relatively uniform set of values, which limits their applicability to tasks that require understanding of nuanced human perspectives. Recent research has underscored the importance of enabling LLMs to support steerable pluralism — the …

RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning

Authors: Aleksei Arzhantsev, Otmane Sakhi, Flavian Vasile Published: 2025-10-03 arXiv: 2510.02892v1 PDF: Download

Abstract: Reinforcement learning (RL) is central to improving reasoning in large language models (LLMs) but typically requires ground-truth rewards. Test-Time Reinforcement Learning (TTRL) removes this need by using majority-vote rewards, but relies on heavy online RL and incurs substantial computational cost…

The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models

Authors: Phuc Minh Nguyen, Chinh D. La, Duy M. H. Nguyen et al. Published: 2025-10-02 arXiv: 2510.02230v1 PDF: Download

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key method for improving Large Language Models’ reasoning capabilities, yet recent evidence suggests it may paradoxically shrink the reasoning boundary rather than expand it. This paper investigates the shrinkage issue of RLVR by…

Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning

Authors: Claudio Fanconi, Nicolás Astorga, Mihaela van der Schaar Published: 2025-10-02 arXiv: 2510.01857v1 PDF: Download

Abstract: We reframe and operationalise adversarial inverse reinforcement learning (IRL) to large language model reasoning, learning a dense, token-level reward model for process supervision directly from expert demonstrations rather than imitating style via supervised fine-tuning. The learned reasoning rewar…

Eliciting Chain-of-Thought Reasoning for Time Series Analysis using Reinforcement Learning

Authors: Felix Parker, Nimeesha Chan, Chi Zhang et al. Published: 2025-10-01 arXiv: 2510.01116v1 PDF: Download

Abstract: Complex numerical time series analysis often demands multi-step reasoning capabilities beyond current models’ reach. Tasks like medical diagnosis and weather forecasting require sequential reasoning processes — including counterfactual analysis, logical deduction, knowledge application, and multi-m…

DeepCodeSeek: Real-Time API Retrieval for Context-Aware Code Generation

Authors: Esakkivel Esakkiraja, Denis Akhiyarov, Aditya Shanmugham et al. Published: 2025-09-30 arXiv: 2509.25716v1 PDF: Download

Abstract: Current search techniques are limited to standard RAG query-document applications. In this paper, we propose a novel technique to expand the code and index for predicting the required APIs, directly enabling high-quality, end-to-end code generation for auto-completion and agentic AI applications. We…

Geo-R1: Unlocking VLM Geospatial Reasoning with Cross-View Reinforcement Learning

Authors: Chenhui Xu, Fuxun Yu, Michael J. Bianco et al. Published: 2025-09-29 arXiv: 2510.00072v1 PDF: Download

Abstract: We introduce Geo-R1, a reasoning-centric post-training framework that unlocks geospatial reasoning in vision-language models by combining thinking scaffolding and elevating. In the scaffolding stage, Geo-R1 instills a “geospatial thinking paradigm” via supervised fine-tuning on synthetic chain-of-t…

Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models

Authors: Guanxu Chen, Yafu Li, Yuxian Jiang et al. Published: 2025-09-28 arXiv: 2509.23962v1 PDF: Download

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) for large language models (LLMs) has achieved remarkable progress in enhancing LLMs’ reasoning capabilities on tasks with clear correctness criteria, such as mathematical reasoning tasks. Several training metrics, such as entropy or response leng…

Causally-Enhanced Reinforcement Policy Optimization

Authors: Xiangqi Wang, Yue Huang, Yujun Zhou et al. Published: 2025-09-27 arXiv: 2509.23095v1 PDF: Download

Abstract: Large language models (LLMs) trained with reinforcement objectives often achieve superficially correct answers via shortcut strategies, pairing correct outputs with spurious or unfaithful reasoning and degrading under small causal perturbations. We introduce Causally-Enhanced Policy Optimization (CE…

Tagging the Thought: Unlocking Personalization Reasoning via Reinforcement Learning

Authors: Song Jin, Juntian Zhang, Yong Liu et al. Published: 2025-09-27 arXiv: 2509.23140v1 PDF: Download

Abstract: Recent advancements have endowed Large Language Models (LLMs) with impressive general reasoning capabilities, yet they often struggle with personalization reasoning - the crucial ability to analyze user history, infer unique preferences, and generate tailored responses. To address this limitation, w…

Reinforcement Learning-Guided Chain-of-Draft for Token-Efficient Code Generation

Authors: Xunzhu Tang, Iyiola Emmanuel Olatunji, Tiezhu Sun et al. Published: 2025-09-26 arXiv: 2509.25243v1 PDF: Download

Abstract: LLMs demonstrate surface-level fluency in code generation but struggle with structured reasoning tasks requiring correctness and semantic alignment. While Chain-of-Thought (CoT) prompting enhances reasoning through intermediate steps, it suffers from verbosity and inefficiency. Chain-of-Draft (CoD) …

Expanding Reasoning Potential in Foundation Model by Learning Diverse Chains of Thought Patterns

Authors: Xuemiao Zhang, Can Ren, Chengying Tu et al. Published: 2025-09-25 arXiv: 2509.21124v2 PDF: Download

Abstract: Recent progress in large reasoning models for challenging mathematical reasoning has been driven by reinforcement learning (RL). Incorporating long chain-of-thought (CoT) data during mid-training has also been shown to substantially improve reasoning depth. However, current approaches often utilize …

Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

Authors: Keliang Liu, Dingkang Yang, Ziyun Qian et al. Published: 2025-09-20 arXiv: 2509.16679v1 PDF: Download

Abstract: In recent years, training methods centered on Reinforcement Learning (RL) have markedly enhanced the reasoning and alignment performance of Large Language Models (LLMs), particularly in understanding human intents, following user instructions, and bolstering inferential strength. Although existing s…

Fleming-R1: Toward Expert-Level Medical Reasoning via Reinforcement Learning

Authors: Chi Liu, Derek Li, Yan Shu et al. Published: 2025-09-18 arXiv: 2509.15279v1 PDF: Download

Abstract: While large language models show promise in medical applications, achieving expert-level clinical reasoning remains challenging due to the need for both accurate answers and transparent reasoning processes. To address this challenge, we introduce Fleming-R1, a model designed for verifiable medical r…

RationAnomaly: Log Anomaly Detection with Rationality via Chain-of-Thought and Reinforcement Learning

Authors: Song Xu, Yilun Liu, Minggui He et al. Published: 2025-09-18 arXiv: 2509.14693v2 PDF: Download

Abstract: Logs constitute a form of evidence signaling the operational status of software systems. Automated log anomaly detection is crucial for ensuring the reliability of modern software systems. However, existing approaches face significant limitations: traditional deep learning models lack interpretabili…

Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models

Authors: Yan Chen, Long Li, Teng Xi et al. Published: 2025-09-16 arXiv: 2509.13031v2 PDF: Download

Abstract: Reinforcement learning (RL) has proven highly effective in eliciting the reasoning capabilities of large language models (LLMs). Inspired by this success, recent studies have explored applying similar techniques to vision-language models (VLMs), aiming to enhance their reasoning performance. However…

Narrative-Guided Reinforcement Learning: A Platform for Studying Language Model Influence on Decision Making

Authors: Anup Tuladhar, Araz Minhas, Adam Kirton et al. Published: 2025-09-10 arXiv: 2509.08785v1 PDF: Download

Abstract: We present a preliminary experimental platform that explores how narrative elements might shape AI decision-making by combining reinforcement learning (RL) with language model reasoning. While AI systems can now both make decisions and engage in narrative reasoning, these capabilities have mostly be…

A Survey of Reinforcement Learning for Large Reasoning Models

Authors: Kaiyan Zhang, Yuxin Zuo, Bingxiang He et al. Published: 2025-09-10 arXiv: 2509.08827v3 PDF: Download

Abstract: In this paper, we survey recent advances in Reinforcement Learning (RL) for reasoning with Large Language Models (LLMs). RL has achieved remarkable success in advancing the frontier of LLM capabilities, particularly in addressing complex logical tasks such as mathematics and coding. As a result, RL …

Teaching AI Stepwise Diagnostic Reasoning with Report-Guided Chain-of-Thought Learning

Authors: Yihong Luo, Wenwu He, Zhuo-Xu Cui et al. Published: 2025-09-08 arXiv: 2509.06409v1 PDF: Download

Abstract: This study presents DiagCoT, a multi-stage framework that applies supervised fine-tuning to general-purpose vision-language models (VLMs) to emulate radiologists’ stepwise diagnostic reasoning using only free-text reports. DiagCoT combines contrastive image-report tuning for domain alignment, chain-…

Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models

Authors: Yinjie Wang, Ling Yang, Bowen Li et al. Published: 2025-09-08 arXiv: 2509.06949v1 PDF: Download

Abstract: We propose TraceRL, a trajectory-aware reinforcement learning framework for diffusion language models (DLMs) that incorporates preferred inference trajectory into post-training, and is applicable across different architectures. Equipped with a diffusion-based value model that enhances training stabi…

Proof2Silicon: Prompt Repair for Verified Code and Hardware Generation via Reinforcement Learning

Authors: Manvi Jha, Jiaxin Wan, Deming Chen Published: 2025-09-07 arXiv: 2509.06239v2 PDF: Download

Abstract: Large Language Models (LLMs) have demonstrated impressive capabilities in automated code generation but frequently produce code that fails formal verification, an essential requirement for hardware and safety-critical domains. To overcome this fundamental limitation, we previously proposed PREFACE, …

Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL

Authors: Haoyang He, Zihua Rong, Kun Ji et al. Published: 2025-09-07 arXiv: 2509.06024v1 PDF: Download

Abstract: Reinforcement learning (RL) has recently become the dominant paradigm for strengthening the reasoning abilities of large language models (LLMs). Yet the rule-based reward functions commonly used on mathematical or programming benchmarks assess only answer format and correctness, providing no signal …

SharedRep-RLHF: A Shared Representation Approach to RLHF with Diverse Preferences

Authors: Arpan Mukherjee, Marcello Bullo, Deniz Gündüz Published: 2025-09-03 arXiv: 2509.03672v1 PDF: Download

Abstract: Uniform-reward reinforcement learning from human feedback (RLHF), which trains a single reward model to represent the preferences of all annotators, fails to capture the diversity of opinions across sub-populations, inadvertently favoring dominant groups. The state-of-the-art, MaxMin-RLHF, addresses…

Think in Games: Learning to Reason in Games via Reinforcement Learning with Large Language Models

Authors: Yi Liao, Yu Gu, Yuan Sui et al. Published: 2025-08-29 arXiv: 2508.21365v1 PDF: Download

Abstract: Large language models (LLMs) excel at complex reasoning tasks such as mathematics and coding, yet they frequently struggle with simple interactive tasks that young children perform effortlessly. This discrepancy highlights a critical gap between declarative knowledge (knowing about something) and pr…

Learning to Generate Unit Test via Adversarial Reinforcement Learning

Authors: Dongjun Lee, Changho Hwang, Kimin Lee Published: 2025-08-28 arXiv: 2508.21107v2 PDF: Download

Abstract: Unit testing is a core practice in programming, enabling systematic evaluation of programs produced by human developers or large language models (LLMs). Given the challenges in writing comprehensive unit tests, LLMs have been employed to automate test generation, yet methods for training LLMs to pro…

Learning Game-Playing Agents with Generative Code Optimization

Authors: Zhiyi Kuang, Ryan Rong, YuCheng Yuan et al. Published: 2025-08-27 arXiv: 2508.19506v1 PDF: Download

Abstract: We present a generative optimization approach for learning game-playing agents, where policies are represented as Python programs and refined using large language models (LLMs). Our method treats decision-making policies as self-evolving code, with current observation as input and an in-game action …

AR$^2$: Adversarial Reinforcement Learning for Abstract Reasoning in Large Language Models

Authors: Cheng-Kai Yeh, Hsing-Wang Lee, Chung-Hung Kuo et al. Published: 2025-08-27 arXiv: 2509.03537v1 PDF: Download

Abstract: Abstraction—the ability to recognize and distill essential computational patterns from complex problem statements—is a foundational skill in computer science, critical both for human problem-solvers and coding-oriented large language models (LLMs). Despite recent advances in training LLMs for code…

VERIRL: Boosting the LLM-based Verilog Code Generation via Reinforcement Learning

Authors: Fu Teng, Miao Pan, Xuhong Zhang et al. Published: 2025-08-25 arXiv: 2508.18462v1 PDF: Download

Abstract: Recent advancements in code generation have shown remarkable success across software domains, yet hardware description languages (HDLs) such as Verilog remain underexplored due to their concurrency semantics, syntactic rigidity, and simulation complexity. In this work, we address these challenges by…

ChartMaster: Advancing Chart-to-Code Generation with Real-World Charts and Chart Similarity Reinforcement Learning

Authors: Wentao Tan, Qiong Cao, Chao Xue et al. Published: 2025-08-25 arXiv: 2508.17608v2 PDF: Download

Abstract: The chart-to-code generation task requires MLLMs to convert chart images into executable code. This task faces two main challenges: limited data diversity and the difficulty of maintaining visual consistency between generated charts and the original ones. Existing datasets mainly rely on synthetic s…

KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF

Authors: Jason R Brown, Lennie Wells, Edward James Young et al. Published: 2025-08-23 arXiv: 2508.17000v1 PDF: Download

Abstract: Proximal Policy Optimisation (PPO) is an established and effective policy gradient algorithm used for Language Model Reinforcement Learning from Human Feedback (LM-RLHF). PPO performs well empirically but has a heuristic motivation and handles the KL-divergence constraint used in LM-RLHF in an ad-ho…

CARFT: Boosting LLM Reasoning via Contrastive Learning with Annotated Chain-of-Thought-based Reinforced Fine-Tuning

Authors: Wenqiao Zhu, Ji Liu, Rongjuncheng Zhang et al. Published: 2025-08-21 arXiv: 2508.15868v2 PDF: Download

Abstract: Reasoning capability plays a significantly critical role in the the broad applications of Large Language Models (LLMs). To enhance the reasoning performance of LLMs, diverse Reinforcement Learning (RL)-based fine-tuning approaches have been proposed to address the limited generalization capability o…

Breaking the SFT Plateau: Multimodal Structured Reinforcement Learning for Chart-to-Code Generation

Authors: Lei Chen, Xuanle Zhao, Zhixiong Zeng et al. Published: 2025-08-19 arXiv: 2508.13587v1 PDF: Download

Abstract: While reinforcement learning (RL) has proven highly effective for general reasoning in vision-language models, its application to tasks requiring in-depth understanding of information-rich images and generation of structured outputs remains underexplored. Chart-to-code generation exemplifies this ch…

Legal$Δ$: Enhancing Legal Reasoning in LLMs via Reinforcement Learning with Chain-of-Thought Guided Information Gain

Authors: Xin Dai, Buqiang Xu, Zhenghao Liu et al. Published: 2025-08-17 arXiv: 2508.12281v2 PDF: Download

Abstract: Legal Artificial Intelligence (LegalAI) has achieved notable advances in automating judicial decision-making with the support of Large Language Models (LLMs). However, existing legal LLMs still struggle to generate reliable and interpretable reasoning processes. They often default to fast-thinking b…

ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Models

Authors: Yuanfeng Xu, Zehui Dai, Jian Liang et al. Published: 2025-08-17 arXiv: 2508.12387v1 PDF: Download

Abstract: Small Language Models (SLMs) are a cost-effective alternative to Large Language Models (LLMs), but often struggle with complex reasoning due to their limited capacity and a tendency to produce mistakes or inconsistent answers during multi-step reasoning. Existing efforts have improved SLM performanc…

From Intent to Execution: Multimodal Chain-of-Thought Reinforcement Learning for Precise CAD Code Generation

Authors: Ke Niu, Haiyang Yu, Zhuofan Chen et al. Published: 2025-08-13 arXiv: 2508.10118v2 PDF: Download

Abstract: Computer-Aided Design (CAD) plays a vital role in engineering and manufacturing, yet current CAD workflows require extensive domain expertise and manual modeling effort. Recent advances in large language models (LLMs) have made it possible to generate code from natural language, opening new opportun…

Bridging Formal Language with Chain-of-Thought Reasoning to Geometry Problem Solving

Authors: Tianyun Yang, Yunwen Li, Ziniu Li et al. Published: 2025-08-12 arXiv: 2508.09099v1 PDF: Download

Abstract: Large vision language models exhibit notable limitations on Geometry Problem Solving (GPS) because of their unreliable diagram interpretation and pure natural-language reasoning. A recent line of work mitigates this by using symbolic solvers: the model directly generates a formal program that a geom…

Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learning

Authors: Shu Wu, Chenxing Li, Wenfu Wang et al. Published: 2025-08-11 arXiv: 2508.08039v3 PDF: Download

Abstract: Recent advancements in large language models, multimodal large language models, and large audio language models (LALMs) have significantly improved their reasoning capabilities through reinforcement learning with rule-based rewards. However, the explicit reasoning process has yet to show significant…

Klear-CodeTest: Scalable Test Case Generation for Code Reinforcement Learning

Authors: Jia Fu, Xinyu Yang, Hongzhi Zhang et al. Published: 2025-08-07 arXiv: 2508.05710v2 PDF: Download

Abstract: Precise, correct feedback is crucial for effectively training large language models (LLMs) in code reinforcement learning. However, synthesizing high-quality test cases remains a profoundly challenging and unsolved problem. In this work, we present Klear-CodeTest, a comprehensive test case synthesis…

Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling

Authors: Bingguang Hao, Zengzhuang Xu, Maolin Wang et al. Published: 2025-08-07 arXiv: 2508.05118v4 PDF: Download

Abstract: The effective training of Large Language Models (LLMs) for function calling faces a critical challenge: balancing exploration of complex reasoning paths with stable policy optimization. Standard methods like Supervised Fine-Tuning (SFT) fail to instill robust reasoning, and traditional Reinforcement…

Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning

Authors: Chang Tian, Matthew B. Blaschko, Mingzhe Xing et al. Published: 2025-08-06 arXiv: 2508.04848v1 PDF: Download

Abstract: Reinforcement learning (RL) has become a key technique for enhancing the reasoning abilities of large language models (LLMs), with policy-gradient algorithms dominating the post-training stage because of their efficiency and effectiveness. However, most existing benchmarks evaluate large-language-mo…

AttriLens-Mol: Attribute Guided Reinforcement Learning for Molecular Property Prediction with Large Language Models

Authors: Xuan Lin, Long Chen, Yile Wang Published: 2025-08-06 arXiv: 2508.04748v2 PDF: Download

Abstract: Large Language Models (LLMs) have shown promise in assisting molecular property prediction tasks but often rely on human-crafted prompts and chain-of-thought templates. While recent advanced large reasoning models like DeepSeek-R1 employ reinforcement learning for an extended “thinking” process, t…

G-Core: A Simple, Scalable and Balanced RLHF Trainer

Authors: Junyu Wu, Weiming Chang, Xiaotao Liu et al. Published: 2025-07-30 arXiv: 2507.22789v2 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) has become an increasingly popular paradigm for training large language models (LLMs) and diffusion models. While existing RLHF training systems have enabled significant progress, they often face challenges in scaling to multi-modal and diffusion wor…

Post-Training Large Language Models via Reinforcement Learning from Self-Feedback

Authors: Carel van Niekerk, Renato Vukovic, Benjamin Matthias Ruppik et al. Published: 2025-07-29 arXiv: 2507.21931v1 PDF: Download

Abstract: Large Language Models (LLMs) often produce plausible but poorly-calibrated answers, limiting their reliability on reasoning-intensive tasks. We present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that uses the model’s own confidence as an intrinsic reward, mimicking how h…

UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models’ Reasoning Abilities

Authors: Dong Du, Shulin Liu, Tao Yang et al. Published: 2025-07-26 arXiv: 2507.19766v1 PDF: Download

Abstract: Recent advances in large language models (LLMs) have highlighted the potential of reinforcement learning with verifiable rewards (RLVR) to enhance reasoning capabilities through extended output sequences. However, traditional RL frameworks face inefficiencies when handling ultra-long outputs due to …

Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning

Authors: Ang Li, Charles Wang, Deqing Fu et al. Published: 2025-07-22 arXiv: 2507.16746v2 PDF: Download

Abstract: Humans often use visual aids, for example diagrams or sketches, when solving complex problems. Training multimodal models to do the same, known as Visual Chain of Thought (Visual CoT), is challenging due to: (1) poor off-the-shelf visual CoT performance, which hinders reinforcement learning, and (2)…

Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning

Authors: Junhao Shen, Haiteng Zhao, Yuzhe Gu et al. Published: 2025-07-22 arXiv: 2507.16814v2 PDF: Download

Abstract: Enhancing large vision-language models (LVLMs) with visual slow-thinking reasoning is crucial for solving complex multimodal tasks. However, since LVLMs are mainly trained with vision-language alignment, it is difficult to adopt on-policy reinforcement learning (RL) to develop the slow thinking abil…

Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasoner

Authors: Lei Chen, Xuanle Zhao, Zhixiong Zeng et al. Published: 2025-07-21 arXiv: 2507.15509v2 PDF: Download

Abstract: Recently, inspired by OpenAI-o1/o3 and Deepseek-R1, the R1-Style method based on reinforcement learning fine-tuning has received widespread attention from the community. Previous R1-Style methods mainly focus on mathematical reasoning and code intelligence. It is of great research significance to ve…

Dr. Boot: Bootstrapping Program Synthesis Language Models to Perform Repairing

Authors: Noah van der Vleuten Published: 2025-07-20 arXiv: 2507.15889v1 PDF: Download

Abstract: Language models for program synthesis are usually trained and evaluated on programming competition datasets (MBPP, APPS). However, these datasets are limited in size and quality, while these language models are extremely data hungry. Additionally, the language models have a misaligned program synthe…

From Roots to Rewards: Dynamic Tree Reasoning with Reinforcement Learning

Authors: Ahmed Bahloul, Simon Malberg Published: 2025-07-17 arXiv: 2507.13142v4 PDF: Download

Abstract: Modern language models address complex questions through chain-of-thought (CoT) reasoning (Wei et al., 2023) and retrieval augmentation (Lewis et al., 2021), yet struggle with error propagation and knowledge integration. Tree-structured reasoning methods, particularly the Probabilistic Tree-of-Thoug…

Fly, Fail, Fix: Iterative Game Repair with Reinforcement Learning and Large Multimodal Models

Authors: Alex Zook, Josef Spjut, Jonathan Tremblay Published: 2025-07-16 arXiv: 2507.12666v1 PDF: Download

Abstract: Game design hinges on understanding how static rules and content translate into dynamic player behavior - something modern generative systems that inspect only a game’s code or assets struggle to capture. We present an automated design iteration framework that closes this gap by pairing a reinforcem…

Thought Purity: A Defense Framework For Chain-of-Thought Attack

Authors: Zihao Xue, Zhen Bi, Long Ma et al. Published: 2025-07-16 arXiv: 2507.12314v2 PDF: Download

Abstract: While reinforcement learning-trained Large Reasoning Models (LRMs, e.g., Deepseek-R1) demonstrate advanced reasoning capabilities in the evolving Large Language Models (LLMs) domain, their susceptibility to security threats remains a critical vulnerability. This weakness is particularly evident in C…

Deep Reinforcement Learning with Gradient Eligibility Traces

Authors: Esraa Elelimy, Brett Daley, Andrew Patterson et al. Published: 2025-07-12 arXiv: 2507.09087v2 PDF: Download

Abstract: Achieving fast and stable off-policy learning in deep reinforcement learning (RL) is challenging. Most existing methods rely on semi-gradient temporal-difference (TD) methods for their simplicity and efficiency, but are consequently susceptible to divergence. While more principled approaches like Gr…

Enhancing RLHF with Human Gaze Modeling

Authors: Karim Galliamov, Ivan Titov, Ilya Pershin Published: 2025-07-11 arXiv: 2507.09016v2 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) aligns language models with human preferences but is computationally expensive. We explore two approaches that leverage human gaze modeling to enhance RLHF: (1) gaze-aware reward models and (2) gaze-based distribution of sparse rewards at token level…

CTRLS: Chain-of-Thought Reasoning via Latent State-Transition

Authors: Junda Wu, Yuxin Xiong, Xintong Li et al. Published: 2025-07-10 arXiv: 2507.08182v1 PDF: Download

Abstract: Chain-of-thought (CoT) reasoning enables large language models (LLMs) to break down complex problems into interpretable intermediate steps, significantly enhancing model transparency and performance in reasoning tasks. However, conventional CoT methods rely on heuristic sampling without structured m…

ChipSeek-R1: Generating Human-Surpassing RTL with LLM via Hierarchical Reward-Driven Reinforcement Learning

Authors: Zhirong Chen, Kaiyan Chang, Zhuolin Li et al. Published: 2025-07-07 arXiv: 2507.04736v1 PDF: Download

Abstract: Large Language Models (LLMs) show significant potential for automating Register-Transfer Level (RTL) code generation. However, current approaches face a critical challenge: they can not simultaneously optimize for functional correctness and hardware quality (Power, Performance, Area - PPA). Methods …

Enhancing Spatial Reasoning in Vision-Language Models via Chain-of-Thought Prompting and Reinforcement Learning

Authors: Binbin Ji, Siddharth Agrawal, Qiance Tang et al. Published: 2025-07-06 arXiv: 2507.13362v1 PDF: Download

Abstract: This study investigates the spatial reasoning capabilities of vision-language models (VLMs) through Chain-of-Thought (CoT) prompting and reinforcement learning. We begin by evaluating the impact of different prompting strategies and find that simple CoT formats, where the model generates a reasoning…

Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess

Authors: Dongyoon Hwang, Hojoon Lee, Jaegul Choo et al. Published: 2025-07-01 arXiv: 2507.00726v3 PDF: Download

Abstract: While reinforcement learning (RL) for large language models (LLMs) has shown promise in mathematical reasoning, strategic reasoning for LLMs using RL remains largely unexplored. We investigate whether LLMs can develop strategic reasoning capabilities through RL in chess. To this end, we leverage a c…

Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning

Authors: Miles Turpin, Andy Arditi, Marvin Li et al. Published: 2025-06-28 arXiv: 2506.22777v2 PDF: Download

Abstract: Language models trained with reinforcement learning (RL) can engage in reward hacking—the exploitation of unintended strategies for high reward—without revealing this behavior in their chain-of-thought reasoning. This makes the detection of reward hacking difficult, posing risks for high-stakes ap…

The Hidden Link Between RLHF and Contrastive Learning

Authors: Xufei Lv, Kehai Chen, Haoyuan Sun et al. Published: 2025-06-27 arXiv: 2506.22578v2 PDF: Download

Abstract: Alignment of large language models (LLMs) with human values has recently garnered significant attention, with prominent examples including the canonical yet costly Reinforcement Learning from Human Feedback (RLHF) and the simple Direct Preference Optimization (DPO). In this work, we demonstrate that…

ReCode: Updating Code API Knowledge with Reinforcement Learning

Authors: Haoze Wu, Yunzhi Yao, Wenhao Yu et al. Published: 2025-06-25 arXiv: 2506.20495v4 PDF: Download

Abstract: Large Language Models (LLMs) exhibit remarkable code generation capabilities but falter when adapting to frequent updates in external library APIs. This critical limitation, stemming from reliance on outdated API knowledge from their training data, even with access to current documentation, impedes …

RecLLM-R1: A Two-Stage Training Paradigm with Reinforcement Learning and Chain-of-Thought v1

Authors: Yu Xie, Xingkai Ren, Ying Qi et al. Published: 2025-06-24 arXiv: 2506.19235v1 PDF: Download

Abstract: Traditional recommendation systems often grapple with “filter bubbles”, underutilization of external knowledge, and a disconnect between model optimization and business policy iteration. To address these limitations, this paper introduces RecLLM-R1, a novel recommendation framework leveraging Large …

ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs

Authors: Jiaru Zou, Ling Yang, Jingwen Gu et al. Published: 2025-06-23 arXiv: 2506.18896v2 PDF: Download

Abstract: Process Reward Models (PRMs) have recently emerged as a powerful framework for supervising intermediate reasoning steps in large language models (LLMs). Previous PRMs are primarily trained on model final output responses and struggle to evaluate intermediate thinking trajectories robustly, especiall…

GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning

Authors: Bo Liu, Xiangyu Zhao, Along He et al. Published: 2025-06-22 arXiv: 2506.17939v2 PDF: Download

Abstract: Medical visual question answering aims to support clinical decision-making by enabling models to answer natural language questions based on medical images. While recent advances in multi-modal learning have significantly improved performance, current methods still suffer from limited answer reliabil…

AutoRule: Reasoning Chain-of-thought Extracted Rule-based Rewards Improve Preference Learning

Authors: Tevin Wang, Chenyan Xiong Published: 2025-06-18 arXiv: 2506.15651v1 PDF: Download

Abstract: Rule-based rewards offer a promising strategy for improving reinforcement learning from human feedback (RLHF), but current approaches often rely on manual rule engineering. We present AutoRule, a fully automated method for extracting rules from preference feedback and formulating them into rule-base…

PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning

Authors: Yizhen Zhang, Yang Ding, Shuoshuo Zhang et al. Published: 2025-06-17 arXiv: 2506.14907v1 PDF: Download

Abstract: Inspired by the impressive reasoning capabilities demonstrated by reinforcement learning approaches like DeepSeek-R1, recent emerging research has begun exploring the use of reinforcement learning (RL) to enhance vision-language models (VLMs) for multimodal reasoning tasks. However, most existing mu…

Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models

Authors: Vaskar Nath, Elaine Lau, Anisha Gunjal et al. Published: 2025-06-16 arXiv: 2506.13923v2 PDF: Download

Abstract: We study the process through which reasoning models trained with reinforcement learning on verifiable rewards (RLVR) can learn to solve new problems. We find that RLVR drives performance in two main ways: (1) by compressing pass@$k$ into pass@1 and (2) via “capability gain” in which models learn to …

Seewo’s Submission to MLC-SLM: Lessons learned from Speech Reasoning Language Models

Authors: Bo Li, Chengben Xu, Wufeng Zhang Published: 2025-06-16 arXiv: 2506.13300v3 PDF: Download

Abstract: This paper presents Seewo’s systems for both tracks of the Multilingual Conversational Speech Language Model Challenge (MLC-SLM), addressing automatic speech recognition (ASR) and speaker diarization with ASR (SD-ASR). We introduce a multi-stage training pipeline that explicitly enhances reasoning a…

Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning

Authors: Shulin Tian, Ruiqi Wang, Hongming Guo et al. Published: 2025-06-16 arXiv: 2506.13654v1 PDF: Download

Abstract: We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e., in days and weeks) egocentric videos, which leverages a structured Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained via reinforcement learning (RL). Inspired by human problem-solving strategies, …

Authors: Jiancong Xiao, Zhekun Shi, Kaizhao Liu et al. Published: 2025-06-14 arXiv: 2506.12350v1 PDF: Download

Abstract: Despite its empirical success, Reinforcement Learning from Human Feedback (RLHF) has been shown to violate almost all the fundamental axioms in social choice theory — such as majority consistency, pairwise majority consistency, and Condorcet consistency. This raises a foundational question: why doe…

Reinforcement learning fine-tuning of language model for instruction following and math reasoning

Authors: Yifu Han, Geo Zhang Published: 2025-06-11 arXiv: 2506.21560v2 PDF: Download

Abstract: This study investigates the effectiveness of reinforcement learning (RL) fine-tuning techniques on a compact language model (Qwen2.5-0.5B Base) for two challenging tasks: instruction following and mathematical reasoning. We compare supervised fine-tuning (SFT), Direct Preference Optimization (DPO) u…

SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning

Authors: Xiao Liang, Zhong-Zhi Li, Yeyun Gong et al. Published: 2025-06-10 arXiv: 2506.08989v1 PDF: Download

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for training large language models (LLMs) on complex reasoning tasks, such as mathematical problem solving. A prerequisite for the scalability of RLVR is a high-quality problem set with precise and verifiable answers. However…

Reinforce LLM Reasoning through Multi-Agent Reflection

Authors: Yurun Yuan, Tengyang Xie Published: 2025-06-10 arXiv: 2506.08379v1 PDF: Download

Abstract: Leveraging more test-time computation has proven to be an effective way to boost the reasoning capabilities of large language models (LLMs). Among various methods, the verify-and-improve paradigm stands out for enabling dynamic solution exploration and feedback incorporation. However, existing appro…

Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints

Authors: Yaswanth Chittepu, Blossom Metevier, Will Schwarzer et al. Published: 2025-06-09 arXiv: 2506.08266v1 PDF: Download

Abstract: Existing approaches to language model alignment often treat safety as a tradeoff against helpfulness, which can lead to unacceptable responses in sensitive domains. To ensure reliable performance in such settings, we propose High-Confidence Safe Reinforcement Learning from Human Feedback (HC-RLHF), …

Gradual Transition from Bellman Optimality Operator to Bellman Operator in Online Reinforcement Learning

Authors: Motoki Omura, Kazuki Ota, Takayuki Osa et al. Published: 2025-06-06 arXiv: 2506.05968v2 PDF: Download

Abstract: For continuous action spaces, actor-critic methods are widely used in online reinforcement learning (RL). However, unlike RL algorithms for discrete actions, which generally model the optimal value function using the Bellman optimality operator, RL algorithms for continuous actions typically model Q…

Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Router

Authors: Chenyang Shao, Xinyang Liu, Yutang Lin et al. Published: 2025-06-06 arXiv: 2506.05901v1 PDF: Download

Abstract: Multi-step reasoning has proven essential for enhancing the problem-solving capabilities of Large Language Models (LLMs) by decomposing complex tasks into intermediate steps, either explicitly or implicitly. Extending the reasoning chain at test time through deeper thought processes or broader explo…

Cross-lingual Collapse: How Language-Centric Foundation Models Shape Reasoning in Large Language Models

Authors: Cheonbok Park, Jeonghoon Kim, Joosung Lee et al. Published: 2025-06-06 arXiv: 2506.05850v2 PDF: Download

Abstract: We identify \textbf{Cross-lingual Collapse}, a systematic drift in which the chain-of-thought (CoT) of a multilingual language model reverts to its dominant pre-training language even when the prompt is expressed in a different language. Recent large language models (LLMs) with reinforcement learnin…

RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models

Authors: Qihang Yan, Xinyu Zhang, Luming Guo et al. Published: 2025-06-03 arXiv: 2506.02726v1 PDF: Download

Abstract: Large Language Models (LLMs) struggle with accuracy, domain-specific reasoning, and interpretability in vertical domains. Traditional preference alignment methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) often overlook the underlying knowledge s…

Reasoning-Table: Exploring Reinforcement Learning for Table Reasoning

Authors: Fangyu Lei, Jinxiang Meng, Yiming Huang et al. Published: 2025-06-02 arXiv: 2506.01710v1 PDF: Download

Abstract: Table reasoning, encompassing tasks such as table question answering, fact verification, and text-to-SQL, requires precise understanding of structured tabular data, coupled with numerical computation and code manipulation for effective inference. Supervised fine-tuning (SFT) approaches have achieved…

SuperRL: Reinforcement Learning with Supervision to Boost Language Model Reasoning

Authors: Yihao Liu, Shuocheng Li, Lang Cao et al. Published: 2025-06-01 arXiv: 2506.01096v2 PDF: Download

Abstract: Large language models are increasingly used for complex reasoning tasks where high-quality offline data such as expert-annotated solutions and distilled reasoning traces are often available. However, in environments with sparse rewards, reinforcement learning struggles to sample successful trajector…

CRScore++: Reinforcement Learning with Verifiable Tool and AI Feedback for Code Review

Authors: Manav Nitin Kapadnis, Atharva Naik, Carolyn Rose Published: 2025-05-30 arXiv: 2506.00296v1 PDF: Download

Abstract: Reinforcement learning (RL) to improve code review comment generation requires handling unstructured outputs, making reinforcement learning (RL) feedback challenging. The two main RL approaches, namely RL with Verifiable Feedback (RLVR) and RL with AI Feedback (RLAIF), offer trade-offs: RLVR provide…

ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models

Authors: Mingjie Liu, Shizhe Diao, Ximing Lu et al. Published: 2025-05-30 arXiv: 2505.24864v1 PDF: Download

Abstract: Recent advances in reasoning-centric language models have highlighted reinforcement learning (RL) as a promising method for aligning models with verifiable rewards. However, it remains contentious whether RL truly expands a model’s reasoning capabilities or merely amplifies high-reward outputs alrea…

Towards Effective Code-Integrated Reasoning

Authors: Fei Bai, Yingqian Min, Beichen Zhang et al. Published: 2025-05-30 arXiv: 2505.24480v1 PDF: Download

Abstract: In this paper, we investigate code-integrated reasoning, where models generate code when necessary and integrate feedback by executing it through a code interpreter. To acquire this capability, models must learn when and how to use external code tools effectively, which is supported by tool-augmente…

Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models

Authors: Junyi Li, Hwee Tou Ng Published: 2025-05-30 arXiv: 2505.24630v2 PDF: Download

Abstract: Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning signif…

Accelerating RLHF Training with Reward Variance Increase

Authors: Zonglin Yang, Zhexuan Gu, Houduo Qi et al. Published: 2025-05-29 arXiv: 2505.23247v2 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) is an essential technique for ensuring that large language models (LLMs) are aligned with human values and preferences during the post-training phase. As an effective RLHF approach, group relative policy optimization (GRPO) has demonstrated success i…

Thompson Sampling in Online RLHF with General Function Approximation

Authors: Songtao Feng, Jie Fu Published: 2025-05-29 arXiv: 2505.23927v1 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) has achieved great empirical success in aligning large language models (LLMs) with human preference, and it is of great importance to study the statistical efficiency of RLHF algorithms from a theoretical perspective. In this work, we consider the on…

Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Models

Authors: Zeyu Liu, Yuhang Liu, Guanghao Zhu et al. Published: 2025-05-29 arXiv: 2505.23091v3 PDF: Download

Abstract: Recent advancements in large language models (LLMs) have demonstrated substantial progress in reasoning capabilities, such as DeepSeek-R1, which leverages rule-based reinforcement learning to enhance logical reasoning significantly. However, extending these achievements to multimodal large language …

Fortune: Formula-Driven Reinforcement Learning for Symbolic Table Reasoning in Language Models

Authors: Lang Cao, Jingxian Xu, Hanbing Liu et al. Published: 2025-05-29 arXiv: 2505.23667v2 PDF: Download

Abstract: Tables are a fundamental structure for organizing and analyzing data, making effective table understanding a critical capability for intelligent systems. While large language models (LMs) demonstrate strong general reasoning abilities, they continue to struggle with accurate numerical or symbolic re…

Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start

Authors: Lai Wei, Yuting Li, Kaipeng Zheng et al. Published: 2025-05-28 arXiv: 2505.22334v2 PDF: Download

Abstract: Recent advancements in large language models (LLMs) have demonstrated impressive chain-of-thought reasoning capabilities, with reinforcement learning (RL) playing a crucial role in this progress. While “aha moment” patterns—where models exhibit self-correction through reflection—are often attribut…

Learning a Pessimistic Reward Model in RLHF

Authors: Yinglun Xu, Hangoo Kang, Tarun Suresh et al. Published: 2025-05-26 arXiv: 2505.20556v1 PDF: Download

Abstract: This work proposes `PET’, a novel pessimistic reward fine-tuning method, to learn a pessimistic reward model robust against reward hacking in offline reinforcement learning from human feedback (RLHF). Traditional reward modeling techniques in RLHF train an imperfect reward model, on which a KL regul…

Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO

Authors: Ruizhe Shi, Minhak Song, Runlong Zhou et al. Published: 2025-05-26 arXiv: 2505.19770v2 PDF: Download

Abstract: We present a fine-grained theoretical analysis of the performance gap between reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) under a representation gap. Our study decomposes this gap into two sources: an explicit representation gap under exact optimization…

Interleaved Reasoning for Large Language Models via Reinforcement Learning

Authors: Roy Xie, David Qiu, Deepak Gopinath et al. Published: 2025-05-26 arXiv: 2505.19640v1 PDF: Download

Abstract: Long chain-of-thought (CoT) significantly enhances large language models’ (LLM) reasoning capabilities. However, the extensive reasoning traces lead to inefficiencies and an increased time-to-first-token (TTFT). We propose a novel training paradigm that uses reinforcement learning (RL) to guide reas…

Authors: Jiayi Zhou, Jiaming Ji, Boyuan Chen et al. Published: 2025-05-24 arXiv: 2505.18531v1 PDF: Download

Abstract: Training multi-modal large language models (MLLMs) that align with human intentions is a long-term challenge. Traditional score-only reward models for alignment suffer from low accuracy, weak generalization, and poor interpretability, blocking the progress of alignment methods, e.g., reinforcement l…

Hybrid Latent Reasoning via Reinforcement Learning

Authors: Zhenrui Yue, Bowen Jin, Huimin Zeng et al. Published: 2025-05-24 arXiv: 2505.18454v2 PDF: Download

Abstract: Recent advances in large language models (LLMs) have introduced latent reasoning as a promising alternative to autoregressive reasoning. By performing internal computation with hidden states from previous steps, latent reasoning benefit from more informative features rather than sampling a discrete …

Reward Model Overoptimisation in Iterated RLHF

Authors: Lorenz Wolf, Robert Kirk, Mirco Musolesi Published: 2025-05-23 arXiv: 2505.18126v2 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) is a widely used method for aligning large language models with human preferences. However, RLHF often suffers from reward model overoptimisation, in which models overfit to the reward function, resulting in non-generalisable policies that exploit th…

Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models

Authors: Zekai Zhao, Qi Liu, Kun Zhou et al. Published: 2025-05-23 arXiv: 2505.17697v1 PDF: Download

Abstract: Despite the remarkable reasoning performance, eliciting the long chain-of-thought (CoT) ability in large language models (LLMs) typically requires costly reinforcement learning or supervised fine-tuning on high-quality distilled data. We investigate the internal mechanisms behind this capability and…

Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO

Authors: Chengzhuo Tong, Ziyu Guo, Renrui Zhang et al. Published: 2025-05-22 arXiv: 2505.17017v2 PDF: Download

Abstract: Recent advancements underscore the significant role of Reinforcement Learning (RL) in enhancing the Chain-of-Thought (CoT) reasoning capabilities of large language models (LLMs). Two prominent RL algorithms, Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO), are cent…

ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models

Authors: Razvan-Gabriel Dumitru, Darius Peteleaza, Vikas Yadav et al. Published: 2025-05-22 arXiv: 2505.17250v1 PDF: Download

Abstract: Large language models excel at complex tasks by breaking down problems into structured reasoning steps. However, reasoning traces often extend beyond reaching a correct answer, causing wasted computation, reduced readability, and hallucinations. To address this, we introduce a novel hyperparameter-f…

Effective Reinforcement Learning for Reasoning in Language Models

Authors: Lianghuan Huang, Shuo Li, Sagnik Anupam et al. Published: 2025-05-22 arXiv: 2505.17218v1 PDF: Download

Abstract: Reinforcement learning (RL) has emerged as a promising strategy for improving the reasoning capabilities of language models (LMs) in domains such as mathematics and coding. However, most modern RL algorithms were designed to target robotics applications, which differ significantly from LM reasoning…

Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models

Authors: Jiaqi Wang, Kevin Qinghong Lin, James Cheng et al. Published: 2025-05-22 arXiv: 2505.16854v3 PDF: Download

Abstract: Reinforcement Learning (RL) has proven to be an effective post-training strategy for enhancing reasoning in vision-language models (VLMs). Group Relative Policy Optimization (GRPO) is a recent prominent method that encourages models to generate complete reasoning traces before answering, leading to …

NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning

Authors: Wei Liu, Siya Qi, Xinyu Wang et al. Published: 2025-05-21 arXiv: 2505.16022v2 PDF: Download

Abstract: Recent advances such as DeepSeek R1-Zero highlight the effectiveness of incentive training, a reinforcement learning paradigm that computes rewards solely based on the final answer part of a language model’s output, thereby encouraging the generation of intermediate reasoning steps. However, these m…

Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning

Authors: Haozhe Wang, Alex Su, Weiming Ren et al. Published: 2025-05-21 arXiv: 2505.15966v3 PDF: Download

Abstract: Chain-of-thought reasoning has significantly improved the performance of Large Language Models (LLMs) across various domains. However, this reasoning process has been confined exclusively to textual space, limiting its effectiveness in visually intensive tasks. To address this limitation, we introdu…

RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning

Authors: Qianyue Hao, Sibo Li, Jian Yuan et al. Published: 2025-05-20 arXiv: 2505.14140v2 PDF: Download

Abstract: Despite rapid advancements in large language models (LLMs), the token-level autoregressive nature constrains their complex reasoning capabilities. To enhance LLM reasoning, inference-time techniques, including Chain/Tree/Graph-of-Thought(s), successfully improve the performance, as they are fairly c…

Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuning

Authors: Haolei Xu, Yuchen Yan, Yongliang Shen et al. Published: 2025-05-20 arXiv: 2505.14684v2 PDF: Download

Abstract: Large language models (LLMs) have achieved remarkable progress on mathematical tasks through Chain-of-Thought (CoT) reasoning. However, existing mathematical CoT datasets often suffer from Thought Leaps due to experts omitting intermediate steps, which negatively impacts model learning and generaliz…

Dual-Agent Reinforcement Learning for Automated Feature Generation

Authors: Wanfu Gao, Zengyao Man, Hanlin Pan et al. Published: 2025-05-19 arXiv: 2505.12628v1 PDF: Download

Abstract: Feature generation involves creating new features from raw data to capture complex relationships among the original features, improving model robustness and machine learning performance. Current methods using reinforcement learning for feature generation have made feature exploration more flexible a…

Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learning

Authors: Yansong Ning, Wei Li, Jun Fang et al. Published: 2025-05-17 arXiv: 2505.11827v2 PDF: Download

Abstract: Compressing long chain-of-thought (CoT) from large language models (LLMs) is an emerging strategy to improve the reasoning efficiency of LLMs. Despite its promising benefits, existing studies equally compress all thoughts within a long CoT, hindering more concise and effective reasoning. To this end…

AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning

Authors: Chenwei Lou, Zewei Sun, Xinnian Liang et al. Published: 2025-05-17 arXiv: 2505.11896v2 PDF: Download

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities but often face challenges with tasks requiring sophisticated reasoning. While Chain-of-Thought (CoT) prompting significantly enhances reasoning, it indiscriminately generates lengthy reasoning steps for all queries, leading to su…

Retrospex: Language Agent Meets Offline Reinforcement Learning Critic

Authors: Yufei Xiang, Yiqun Shen, Yeqin Zhang et al. Published: 2025-05-17 arXiv: 2505.11807v2 PDF: Download

Abstract: Large Language Models (LLMs) possess extensive knowledge and commonsense reasoning capabilities, making them valuable for creating powerful agents. However, existing LLM agent frameworks have not fully utilized past experiences for improvement. This work introduces a new LLM-based agent framework ca…

Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models

Authors: Zemin Huang, Zhiyang Chen, Zijun Wang et al. Published: 2025-05-15 arXiv: 2505.10446v3 PDF: Download

Abstract: We introduce the Diffusion Chain of Lateral Thought (DCoLT), a reasoning framework for diffusion language models. DCoLT treats each intermediate step in the reverse diffusion process as a latent “thinking” action and optimizes the entire reasoning trajectory to maximize the reward on the correctness…

Making Small Language Models Efficient Reasoners: Intervention, Supervision, Reinforcement

Authors: Xuechen Zhang, Zijian Huang, Chenshun Ni et al. Published: 2025-05-12 arXiv: 2505.07961v3 PDF: Download

Abstract: Recent research enhances language model reasoning by scaling test-time compute via longer chain-of-thought traces. This often improves accuracy but also introduces redundancy and high computational cost, especially for small language models distilled with supervised fine-tuning (SFT). In this work, …

S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models

Authors: Muzhi Dai, Chenxu Yang, Qingyi Si Published: 2025-05-12 arXiv: 2505.07686v2 PDF: Download

Abstract: As Test-Time Scaling emerges as an active research focus in the large language model community, advanced post-training methods increasingly emphasize extending chain-of-thought (CoT) generation length, thereby enhancing reasoning capabilities to approach Deepseek R1-like reasoning models. However, r…

INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning

Authors: Prime Intellect Team, Sami Jaghouar, Justus Mattern et al. Published: 2025-05-12 arXiv: 2505.07291v1 PDF: Download

Abstract: We introduce INTELLECT-2, the first globally distributed reinforcement learning (RL) training run of a 32 billion parameter language model. Unlike traditional centralized training efforts, INTELLECT-2 trains a reasoning model using fully asynchronous RL across a dynamic, heterogeneous swarm of permi…

Reinforcement Learning under State and Outcome Uncertainty: A Foundational Distributional Perspective

Authors: Larry Preuett, Qiuyi Zhang, Muhammad Aurangzeb Ahmad Published: 2025-05-10 arXiv: 2505.06518v2 PDF: Download

Abstract: In many real-world planning tasks, agents must tackle uncertainty about the environment’s state and variability in the outcomes of any chosen policy. We address both forms of uncertainty as a first step toward safer algorithms in partially observable settings. Specifically, we extend Distributional …

Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models

Authors: Yunxin Li, Zhenyu Liu, Zitao Li et al. Published: 2025-05-08 arXiv: 2505.04921v2 PDF: Download

Abstract: Reasoning lies at the heart of intelligence, shaping the ability to make decisions, draw conclusions, and generalize across domains. In artificial intelligence, as systems increasingly operate in open, uncertain, and multimodal environments, reasoning becomes essential for enabling robust and adapti…

Policy-labeled Preference Learning: Is Preference Enough for RLHF?

Authors: Taehyun Cho, Seokhun Ju, Seungyub Han et al. Published: 2025-05-06 arXiv: 2505.06273v2 PDF: Download

Abstract: To design rewards that align with human goals, Reinforcement Learning from Human Feedback (RLHF) has emerged as a prominent technique for learning reward functions from human preferences and optimizing policies via reinforcement learning algorithms. However, existing RLHF methods often misinterpret …

DYSTIL: Dynamic Strategy Induction with Large Language Models for Reinforcement Learning

Authors: Borui Wang, Kathleen McKeown, Rex Ying Published: 2025-05-06 arXiv: 2505.03209v1 PDF: Download

Abstract: Reinforcement learning from expert demonstrations has long remained a challenging research problem, and existing state-of-the-art methods using behavioral cloning plus further RL training often suffer from poor generalization, low sample efficiency, and poor model interpretability. Inspired by the s…

Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL

Authors: Jiarui Yao, Yifan Hao, Hanning Zhang et al. Published: 2025-05-05 arXiv: 2505.02391v1 PDF: Download

Abstract: Chain-of-thought (CoT) reasoning in large language models (LLMs) can be formalized as a latent variable problem, where the model needs to generate intermediate reasoning steps. While prior approaches such as iterative reward-ranked fine-tuning (RAFT) have relied on such formulations, they typically …

Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math

Authors: Haoran Xu, Baolin Peng, Hany Awadalla et al. Published: 2025-04-30 arXiv: 2504.21233v1 PDF: Download

Abstract: Chain-of-Thought (CoT) significantly enhances formal reasoning capabilities in Large Language Models (LLMs) by training them to explicitly generate intermediate reasoning steps. While LLMs readily benefit from such techniques, improving reasoning in Small Language Models (SLMs) remains challenging d…

Reinforcement Learning for Reasoning in Large Language Models with One Training Example

Authors: Yiping Wang, Qing Yang, Zhiyuan Zeng et al. Published: 2025-04-29 arXiv: 2504.20571v3 PDF: Download

Abstract: We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large language models (LLMs). Applying RLVR to the base model Qwen2.5-Math-1.5B, we identify a single example that elevates model perfo…

Reason Like a Radiologist: Chain-of-Thought and Reinforcement Learning for Verifiable Report Generation

Authors: Peiyuan Jing, Kinhei Lee, Zhenxuan Zhang et al. Published: 2025-04-25 arXiv: 2504.18453v1 PDF: Download

Abstract: Radiology report generation is critical for efficiency but current models lack the structured reasoning of experts, hindering clinical trust and explainability by failing to link visual findings to precise anatomical locations. This paper introduces BoxMed-RL, a groundbreaking unified training frame…

Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning

Authors: Shaokun Zhang, Yi Dong, Jieyu Zhang et al. Published: 2025-04-25 arXiv: 2505.00024v2 PDF: Download

Abstract: Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text space. To enhance LLMs’ tool-calling abilities, previous approaches primarily rely on supervised fine-tuning (SFT) with trajectories distilled from stronger models, often re…

Sparks of Tabular Reasoning via Text2SQL Reinforcement Learning

Authors: Josefa Lia Stoisser, Marc Boubnovski Martell, Julien Fauqueur Published: 2025-04-23 arXiv: 2505.00016v2 PDF: Download

Abstract: This work reframes the Text-to-SQL task as a pathway for teaching large language models (LLMs) to reason over and manipulate tabular data—moving beyond the traditional focus on query generation. We propose a two-stage framework that leverages SQL supervision to develop transferable table reasoning …

SARI: Structured Audio Reasoning via Curriculum-Guided Reinforcement Learning

Authors: Cheng Wen, Tingwei Guo, Shuaijiang Zhao et al. Published: 2025-04-22 arXiv: 2504.15900v3 PDF: Download

Abstract: Recent work shows that reinforcement learning(RL) can markedly sharpen the reasoning ability of large language models (LLMs) by prompting them to “think before answering.” Yet whether and how these gains transfer to audio-language reasoning remains largely unexplored. We extend the Group-Relative Po…

Learning Adaptive Parallel Reasoning with Language Models

Authors: Jiayi Pan, Xiuyu Li, Long Lian et al. Published: 2025-04-21 arXiv: 2504.15466v2 PDF: Download

Abstract: Scaling inference-time computation has substantially improved the reasoning capabilities of language models. However, existing methods have significant limitations: serialized chain-of-thought approaches generate overly long outputs, leading to increased latency and exhausted context windows, while …

Reinforcement Learning from Human Feedback

Authors: Nathan Lambert Published: 2025-04-16 arXiv: 2504.12501v3 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) has become an important technical and storytelling tool to deploy the latest machine learning systems. In this book, we hope to give a gentle introduction to the core methods for people with some level of quantitative background. The book starts with…

d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning

Authors: Siyan Zhao, Devaansh Gupta, Qinqing Zheng et al. Published: 2025-04-16 arXiv: 2504.12216v2 PDF: Download

Abstract: Recent large language models (LLMs) have demonstrated strong reasoning capabilities that benefits from online reinforcement learning (RL). These capabilities have primarily been demonstrated within the left-to-right autoregressive (AR) generation paradigm. In contrast, non-autoregressive paradigms b…

ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback

Authors: Tasnia Rahman, Sathish A. P. Kumar, Sumit Jha et al. Published: 2025-04-07 arXiv: 2504.04657v1 PDF: Download

Abstract: Automated Program Repair tools are developed for generating feedback and suggesting a repair method for erroneous code. State of the art (SOTA) code repair methods rely on data-driven approaches and often fail to deliver solution for complicated programming questions. To interpret the natural langua…

A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization

Authors: Wenyuan Xu, Xiaochen Zuo, Chao Xin et al. Published: 2025-04-07 arXiv: 2504.04950v1 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a important paradigm for aligning large language models (LLMs) with human preferences during post-training. This framework typically involves two stages: first, training a reward model on human preference data, followed by optimizing t…

R2Vul: Learning to Reason about Software Vulnerabilities with Reinforcement Learning and Structured Reasoning Distillation

Authors: Martin Weyssow, Chengran Yang, Junkai Chen et al. Published: 2025-04-07 arXiv: 2504.04699v2 PDF: Download

Abstract: Large language models (LLMs) have shown promising performance in software vulnerability detection, yet their reasoning capabilities remain unreliable. We propose R2Vul, a method that combines reinforcement learning from AI feedback (RLAIF) and structured reasoning distillation to teach small code LL…

Concise Reasoning via Reinforcement Learning

Authors: Mehdi Fatemi, Banafsheh Rafiee, Mingjie Tang et al. Published: 2025-04-07 arXiv: 2504.05185v2 PDF: Download

Abstract: Despite significant advancements in large language models (LLMs), a major drawback of reasoning models is their enormous token usage, which increases computational cost, resource requirements, and response time. In this work, we revisit the core principles of reinforcement learning (RL) and, through…

Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models

Authors: Hung Le, Dai Do, Dung Nguyen et al. Published: 2025-04-03 arXiv: 2504.02273v1 PDF: Download

Abstract: Recent advances in fine-tuning large language models (LLMs) with reinforcement learning (RL) have shown promising improvements in complex reasoning tasks, particularly when paired with chain-of-thought (CoT) prompting. However, these successes have been largely demonstrated on large-scale models wit…

Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models

Authors: Huajie Tan, Yuheng Ji, Xiaoshuai Hao et al. Published: 2025-03-26 arXiv: 2503.20752v3 PDF: Download

Abstract: Visual reasoning abilities play a crucial role in understanding complex multimodal data, advancing both domain-specific applications and artificial general intelligence (AGI). Existing methods enhance Vision-Language Models (VLMs) through Chain-of-Thought (CoT) supervised fine-tuning using meticulou…

One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHF

Authors: Xin Cai Published: 2025-03-25 arXiv: 2503.19523v2 PDF: Download

Abstract: In this article, we primarily examine a variety of RL-based and RL-free methods designed to address Reinforcement Learning from Human Feedback (RLHF) and Large Reasoning Models (LRMs). We begin with a concise overview of the typical steps involved in RLHF and LRMs. Next, we reinterpret several RL-ba…

ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning

Authors: Mingyang Chen, Linzhuang Sun, Tianpeng Li et al. Published: 2025-03-25 arXiv: 2503.19470v3 PDF: Download

Abstract: Large Language Models (LLMs) have shown remarkable capabilities in reasoning, exemplified by the success of OpenAI-o1 and DeepSeek-R1. However, integrating reasoning with external search processes remains challenging, especially for complex multi-hop questions requiring multiple retrieval steps. We …

A Shared Low-Rank Adaptation Approach to Personalized RLHF

Authors: Renpu Liu, Peng Wang, Donghao Li et al. Published: 2025-03-24 arXiv: 2503.19201v1 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for aligning artificial intelligence systems with human values, achieving remarkable success in fine-tuning large language models. However, existing RLHF frameworks often assume that human preferences are relatively…

Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models

Authors: Yang Sui, Yu-Neng Chuang, Guanchu Wang et al. Published: 2025-03-20 arXiv: 2503.16419v4 PDF: Download

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in complex tasks. Recent advancements in Large Reasoning Models (LRMs), such as OpenAI o1 and DeepSeek-R1, have further improved performance in System-2 reasoning domains like mathematics and programming by harnessing supervised …

Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning

Authors: Zhaowei Liu, Xin Guo, Fangqi Lou et al. Published: 2025-03-20 arXiv: 2503.16252v2 PDF: Download

Abstract: Reasoning large language models are rapidly evolving across various domains. However, their capabilities in handling complex financial tasks still require in-depth exploration. In this paper, we introduce Fin-R1, a reasoning large language model specifically designed for the financial sector. Fin-R1…

Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Models

Authors: Man Fai Wong, Chee Wei Tan Published: 2025-03-19 arXiv: 2503.15129v1 PDF: Download

Abstract: This paper studies how AI-assisted programming and large language models (LLM) improve software developers’ ability via AI tools (LLM agents) like Github Copilot and Amazon CodeWhisperer, while integrating human feedback to enhance reinforcement learning (RLHF) with crowd-sourced computation to enha…

Applications of Large Language Model Reasoning in Feature Generation

Authors: Dharani Chandra Published: 2025-03-15 arXiv: 2503.11989v2 PDF: Download

Abstract: Large Language Models (LLMs) have revolutionized natural language processing through their state of art reasoning capabilities. This paper explores the convergence of LLM reasoning techniques and feature generation for machine learning tasks. We examine four key reasoning approaches: Chain of Though…

Authors: Yirong Sun, Yanjun Chen Published: 2025-03-13 arXiv: 2503.10177v2 PDF: Download

Abstract: We propose PRISM, a novel framework designed to overcome the limitations of 2D-based Preference-Based Reinforcement Learning (PBRL) by unifying 3D point cloud modeling and future-aware preference refinement. At its core, PRISM adopts a 3D Point Cloud-Language Model (3D-PC-LLM) to mitigate occlusion …

Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models

Authors: Qiguang Chen, Libo Qin, Jinhao Liu et al. Published: 2025-03-12 arXiv: 2503.09567v5 PDF: Download

Abstract: Recent advancements in reasoning with large language models (RLLMs), such as OpenAI-O1 and DeepSeek-R1, have demonstrated their impressive capabilities in complex domains like mathematics and coding. A central factor in their success lies in the application of long chain-of-thought (Long CoT) charac…

Impoola: The Power of Average Pooling for Image-Based Deep Reinforcement Learning

Authors: Raphael Trumpp, Ansgar Schäfftlein, Mirco Theile et al. Published: 2025-03-07 arXiv: 2503.05546v2 PDF: Download

Abstract: As image-based deep reinforcement learning tackles more challenging tasks, increasing model size has become an important factor in improving performance. Recent studies achieved this by focusing on the parameter efficiency of scaled networks, typically using Impala-CNN, a 15-layer ResNet-inspired ne…

On a Connection Between Imitation Learning and RLHF

Authors: Teng Xiao, Yige Yuan, Mingxiao Li et al. Published: 2025-03-07 arXiv: 2503.05079v1 PDF: Download

Abstract: This work studies the alignment of large language models with preference data from an imitation learning perspective. We establish a close theoretical connection between reinforcement learning from human feedback RLHF and imitation learning (IL), revealing that RLHF implicitly performs imitation lea…

L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning

Authors: Pranjal Aggarwal, Sean Welleck Published: 2025-03-06 arXiv: 2503.04697v2 PDF: Download

Abstract: Reasoning language models have shown an uncanny ability to improve performance at test-time by “thinking longer”-that is, by generating longer chain-of-thought sequences and hence using more compute. However, the length of their chain-of-thought reasoning is not controllable, making it impossible …

LLM Misalignment via Adversarial RLHF Platforms

Authors: Erfan Entezami, Ali Naseh Published: 2025-03-04 arXiv: 2503.03039v1 PDF: Download

Abstract: Reinforcement learning has shown remarkable performance in aligning language models with human preferences, leading to the rise of attention towards developing RLHF platforms. These platforms enable users to fine-tune models without requiring any expertise in developing complex machine learning algo…

On Generalization Across Environments In Multi-Objective Reinforcement Learning

Authors: Jayden Teoh, Pradeep Varakantham, Peter Vamplew Published: 2025-03-02 arXiv: 2503.00799v2 PDF: Download

Abstract: Real-world sequential decision-making tasks often require balancing trade-offs between multiple conflicting objectives, making Multi-Objective Reinforcement Learning (MORL) an increasingly prominent field of research. Despite recent advances, existing MORL literature has narrowly focused on performa…

MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning

Authors: Jiazhen Pan, Che Liu, Junde Wu et al. Published: 2025-02-26 arXiv: 2502.19634v2 PDF: Download

Abstract: Reasoning is a critical frontier for advancing medical image analysis, where transparency and trustworthiness play a central role in both clinician trust and regulatory approval. Although Medical Visual Language Models (VLMs) show promise for radiological tasks, most existing VLMs merely produce fin…

Simplify RLHF as Reward-Weighted SFT: A Variational Method

Authors: Yuhao Du, Zhuo Li, Pengyu Cheng et al. Published: 2025-02-16 arXiv: 2502.11026v2 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning Large Language Models (LLMs) with human values. However, RLHF has been continuously challenged by its high complexity in implementation and computation consumption. Even with recent simplifications, such as Direct Preference O…

Provably Efficient Online RLHF with One-Pass Reward Modeling

Authors: Long-Fei Li, Yu-Yang Qian, Peng Zhao et al. Published: 2025-02-11 arXiv: 2502.07193v3 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) has shown remarkable success in aligning Large Language Models (LLMs) with human preferences. Traditional RLHF methods rely on a fixed dataset, which often suffers from limited coverage. To this end, online RLHF has emerged as a promising direction, …

A Survey on Explainable Deep Reinforcement Learning

Authors: Zelei Cheng, Jiahao Yu, Xinyu Xing Published: 2025-02-08 arXiv: 2502.06869v1 PDF: Download

Abstract: Deep Reinforcement Learning (DRL) has achieved remarkable success in sequential decision-making tasks across diverse domains, yet its reliance on black-box neural architectures hinders interpretability, trust, and deployment in high-stakes applications. Explainable Deep Reinforcement Learning (XRL) …

Adaptive Graph of Thoughts: Test-Time Adaptive Reasoning Unifying Chain, Tree, and Graph Structures

Authors: Tushar Pandey, Ara Ghukasyan, Oktay Goktas et al. Published: 2025-02-07 arXiv: 2502.05078v1 PDF: Download

Abstract: Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, yet their performance is highly dependent on the prompting strategy and model scale. While reinforcement learning and fine-tuning have been deployed to boost reasoning, these approaches incur substantial computational …

Training Language Models to Reason Efficiently

Authors: Daman Arora, Andrea Zanette Published: 2025-02-06 arXiv: 2502.04463v4 PDF: Download

Abstract: Scaling model size and training data has led to great advances in the performance of Large Language Models (LLMs). However, the diminishing returns of this approach necessitate alternative methods to improve model capabilities, particularly in tasks requiring advanced reasoning. Large reasoning mode…

Demystifying Long Chain-of-Thought Reasoning in LLMs

Authors: Edward Yeo, Yuxuan Tong, Morry Niu et al. Published: 2025-02-05 arXiv: 2502.03373v1 PDF: Download

Abstract: Scaling inference compute enhances reasoning in large language models (LLMs), with long chains-of-thought (CoTs) enabling strategies like backtracking and error correction. Reinforcement learning (RL) has emerged as a crucial method for developing these capabilities, yet the conditions under which l…

Advancing Reasoning in Large Language Models: Promising Methods and Approaches

Authors: Avinash Patil, Aryan Jadon Published: 2025-02-05 arXiv: 2502.03671v2 PDF: Download

Abstract: Large Language Models (LLMs) have succeeded remarkably in various natural language processing (NLP) tasks, yet their reasoning capabilities remain a fundamental challenge. While LLMs exhibit impressive fluency and factual recall, their ability to perform complex reasoning-spanning logical deduction,…

Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search

Authors: Maohao Shen, Guangtao Zeng, Zhenting Qi et al. Published: 2025-02-04 arXiv: 2502.02508v3 PDF: Download

Abstract: Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains. Recent studies have shown that increasing test-time computation enhances LLMs’ reasoning capabilities. This typically involves extensive sampling at inference time guided by an external LLM verif…

Process-Supervised Reinforcement Learning for Code Generation

Authors: Yufan Ye, Ting Zhang, Wenbin Jiang et al. Published: 2025-02-03 arXiv: 2502.01715v1 PDF: Download

Abstract: Existing reinforcement learning strategies based on outcome supervision have proven effective in enhancing the performance of large language models(LLMs) for code generation. While reinforcement learning based on process supervision has shown great promise in handling multi-step reasoning tasks, its…

Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration

Authors: Mingyu Chen, Yiding Chen, Wen Sun et al. Published: 2025-02-02 arXiv: 2502.00666v3 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal technique for large language model (LLM) alignment. This paper studies the setting of online RLHF and focus on improving sample efficiency. All existing algorithms in online RLHF, whether doing passive exploration or active e…

BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning

Authors: Han Zhong, Yutong Yin, Shenao Zhang et al. Published: 2025-01-31 arXiv: 2501.18858v2 PDF: Download

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks, yet generating reliable reasoning processes remains a significant challenge. We present a unified probabilistic framework that formalizes LLM reasoning through a novel graphical model incorporating lat…

T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling

Authors: Zhenyu Hou, Xin Lv, Rui Lu et al. Published: 2025-01-20 arXiv: 2501.11651v2 PDF: Download

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in complex reasoning tasks. However, existing approaches mainly rely on imitation learning and struggle to achieve effective test-time scaling. While reinforcement learning (RL) holds promise for enabling self-exploration, recent…

RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems?

Authors: Haotian Xu, Xing Wu, Weinong Wang et al. Published: 2025-01-20 arXiv: 2501.11284v1 PDF: Download

Abstract: Can scaling transform reasoning? In this work, we explore the untapped potential of scaling Long Chain-of-Thought (Long-CoT) data to 1000k samples, pioneering the development of a slow-thinking model, RedStar. Through extensive experiments with various LLMs and different sizes, we uncover the ingred…

Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models

Authors: Fengli Xu, Qianyue Hao, Zefang Zong et al. Published: 2025-01-16 arXiv: 2501.09686v3 PDF: Download

Abstract: Language has long been conceived as an essential tool for human reasoning. The breakthrough of Large Language Models (LLMs) has sparked significant research interest in leveraging these models to tackle complex reasoning tasks. Researchers have moved beyond simple autoregressive token generation by …

RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation

Authors: Kaiqu Liang, Haimin Hu, Ryan Liu et al. Published: 2025-01-15 arXiv: 2501.08617v3 PDF: Download

Abstract: While Reinforcement Learning from Human Feedback (RLHF) has shown promise in aligning generative AI, we present empirical evidence that it can also cause severe, systematic misalignment. We hypothesize that this stems from evaluator feedback depending on downstream outcome predictions (foresight) th…

AlgoPilot: Fully Autonomous Program Synthesis Without Human-Written Programs

Authors: Xiaoxin Yin Published: 2025-01-11 arXiv: 2501.06423v1 PDF: Download

Abstract: Program synthesis has traditionally relied on human-provided specifications, examples, or prior knowledge to generate functional algorithms. Existing methods either emulate human-written algorithms or solve specific tasks without generating reusable programmatic logic, limiting their ability to crea…

Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought

Authors: Violet Xiang, Charlie Snell, Kanishk Gandhi et al. Published: 2025-01-08 arXiv: 2501.04682v1 PDF: Download

Abstract: We propose a novel framework, Meta Chain-of-Thought (Meta-CoT), which extends traditional Chain-of-Thought (CoT) by explicitly modeling the underlying reasoning required to arrive at a particular CoT. We present empirical evidence from state-of-the-art models exhibiting behaviors consistent with in-…

Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model

Authors: Yueqin Yin, Shentao Yang, Yujia Xie et al. Published: 2025-01-06 arXiv: 2501.02790v1 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) has been widely adopted to align language models (LMs) with human preference. Prior RLHF works typically take a bandit formulation, which, though intuitive, ignores the sequential nature of LM generation and can suffer from the sparse reward issue. W…

2024

Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey

Authors: Junqiao Wang, Zeng Zhang, Yangfan He et al. Published: 2024-12-29 arXiv: 2412.20367v5 PDF: Download

Abstract: With the rapid evolution of large language models (LLM), reinforcement learning (RL) has emerged as a pivotal technique for code generation and optimization in various domains. This paper presents a systematic survey of the application of RL in code optimization and generation, highlighting its role…

Low-Rank Contextual Reinforcement Learning from Heterogeneous Human Feedback

Authors: Seong Jin Lee, Will Wei Sun, Yufeng Liu Published: 2024-12-27 arXiv: 2412.19436v1 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) has become a cornerstone for aligning large language models with human preferences. However, the heterogeneity of human feedback, driven by diverse individual contexts and preferences, poses significant challenges for reward learning. To address this…

FedRLHF: A Convergence-Guaranteed Federated Framework for Privacy-Preserving and Personalized RLHF

Authors: Flint Xiaofeng Fan, Cheston Tan, Yew-Soon Ong et al. Published: 2024-12-20 arXiv: 2412.15538v2 PDF: Download

Abstract: In the era of increasing privacy concerns and demand for personalized experiences, traditional Reinforcement Learning with Human Feedback (RLHF) frameworks face significant challenges due to their reliance on centralized data. We introduce Federated Reinforcement Learning with Human Feedback (FedRLH…

Offline Reinforcement Learning for LLM Multi-Step Reasoning

Authors: Huaijie Wang, Shibo Hao, Hanze Dong et al. Published: 2024-12-20 arXiv: 2412.16145v2 PDF: Download

Abstract: Improving the multi-step reasoning ability of large language models (LLMs) with offline reinforcement learning (RL) is essential for quickly adapting them to complex tasks. While Direct Preference Optimization (DPO) has shown promise in aligning LLMs with human preferences, it is less suitable for m…

Emma-X: An Embodied Multimodal Action Model with Grounded Chain of Thought and Look-ahead Spatial Reasoning

Authors: Qi Sun, Pengfei Hong, Tej Deep Pala et al. Published: 2024-12-16 arXiv: 2412.11974v2 PDF: Download

Abstract: Traditional reinforcement learning-based robotic control methods are often task-specific and fail to generalize across diverse environments or unseen objects and instructions. Visual Language Models (VLMs) demonstrate strong scene understanding and planning capabilities but lack the ability to gener…

Does RLHF Scale? Exploring the Impacts From Data, Model, and Method

Authors: Zhenyu Hou, Pengfan Du, Yilin Niu et al. Published: 2024-12-08 arXiv: 2412.06000v1 PDF: Download

Abstract: This study explores the scaling properties of Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). Although RLHF is considered an important step in post-training of LLMs, its scaling potential is still largely unknown. We systematically analyze key components in the RLH…

On the Impact of Fine-Tuning on Chain-of-Thought Reasoning

Authors: Elita Lobo, Chirag Agarwal, Himabindu Lakkaraju Published: 2024-11-22 arXiv: 2411.15382v2 PDF: Download

Abstract: Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities that find applications across diverse domains. Despite their impressive performance, recent studies have highlighted the potential for significant enhancements …

An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces

Authors: Alex Beeson, David Ireland, Giovanni Montana Published: 2024-11-17 arXiv: 2411.11088v1 PDF: Download

Abstract: Expanding reinforcement learning (RL) to offline domains generates promising prospects, particularly in sectors where data collection poses substantial challenges or risks. Pivotal to the success of transferring RL offline is mitigating overestimation bias in value estimates for state-action pairs a…

Sharp Analysis for KL-Regularized Contextual Bandits and RLHF

Authors: Heyang Zhao, Chenlu Ye, Quanquan Gu et al. Published: 2024-11-07 arXiv: 2411.04625v2 PDF: Download

Abstract: Reverse-Kullback-Leibler (KL) regularization has emerged to be a predominant technique used to enhance policy optimization in reinforcement learning (RL) and reinforcement learning from human feedback (RLHF), which forces the learned policy to stay close to a reference policy. While the effectivenes…

SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF

Authors: Atoosa Chegini, Hamid Kazemi, Iman Mirzadeh et al. Published: 2024-11-04 arXiv: 2411.01798v1 PDF: Download

Abstract: In Large Language Model (LLM) development, Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning models with human values and preferences. RLHF traditionally relies on the Kullback-Leibler (KL) divergence between the current policy and a frozen initial policy as a reference, whic…

RL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasoner

Authors: Fu-Chieh Chang, Yu-Ting Lee, Hui-Ying Shih et al. Published: 2024-10-31 arXiv: 2410.23912v2 PDF: Download

Abstract: The reasoning abilities of large language models (LLMs) have improved with chain-of-thought (CoT) prompting, allowing models to solve complex tasks stepwise. However, training CoT capabilities requires detailed reasoning data, which is often scarce. The self-taught reasoner (STaR) framework addresse…

FALCON: Feedback-driven Adaptive Long/short-term memory reinforced Coding Optimization system

Authors: Zeyuan Li, Yangfan He, Lewei He et al. Published: 2024-10-28 arXiv: 2410.21349v5 PDF: Download

Abstract: Recently, large language models (LLMs) have achieved significant progress in automated code generation. Despite their strong instruction-following capabilities, these models frequently struggled to align with user intent in coding scenarios. In particular, they were hampered by datasets that lacked …

Understanding and Alleviating Memory Consumption in RLHF for LLMs

Authors: Jin Zhou, Hanmei Yang, Steven et al. Published: 2024-10-21 arXiv: 2410.15651v1 PDF: Download

Abstract: Fine-tuning with Reinforcement Learning with Human Feedback (RLHF) is essential for aligning large language models (LLMs). However, RLHF often encounters significant memory challenges. This study is the first to examine memory usage in the RLHF context, exploring various memory management strategies…

Improve Vision Language Model Chain-of-thought Reasoning

Authors: Ruohong Zhang, Bowen Zhang, Yanghao Li et al. Published: 2024-10-21 arXiv: 2410.16198v1 PDF: Download

Abstract: Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. However, current training recipes lack robust CoT reasoning data, relying on datasets dominated by short annotations with minimal rationales. In this work, we show that tr…

Integrating Large Language Models and Reinforcement Learning for Non-Linear Reasoning

Authors: Yoav Alon, Cristina David Published: 2024-10-17 arXiv: 2410.13501v1 PDF: Download

Abstract: Large Language Models (LLMs) were shown to struggle with long-term planning, which may be caused by the limited way in which they explore the space of possible solutions. We propose an architecture where a Reinforcement Learning (RL) Agent guides an LLM’s space exploration: (1) the Agent has access …

HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning

Authors: Ayano Hiranaka, Shang-Fu Chen, Chieh-Hsin Lai et al. Published: 2024-10-07 arXiv: 2410.05116v3 PDF: Download

Abstract: Controllable generation through Stable Diffusion (SD) fine-tuning aims to improve fidelity, safety, and alignment with human guidance. Existing reinforcement learning from human feedback methods usually rely on predefined heuristic reward functions or pretrained reward models built on large-scale da…

Understanding Reasoning in Chain-of-Thought from the Hopfieldian View

Authors: Lijie Hu, Liang Liu, Shu Yang et al. Published: 2024-10-04 arXiv: 2410.03595v1 PDF: Download

Abstract: Large Language Models have demonstrated remarkable abilities across various tasks, with Chain-of-Thought (CoT) prompting emerging as a key technique to enhance reasoning capabilities. However, existing research primarily focuses on improving performance, lacking a comprehensive framework to explain …

The Perfect Blend: Redefining RLHF with Mixture of Judges

Authors: Tengyu Xu, Eryk Helenowski, Karthik Abinav Sankararaman et al. Published: 2024-09-30 arXiv: 2409.20370v1 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) has become the leading approach for fine-tuning large language models (LLM). However, RLHF has limitations in multi-task learning (MTL) due to challenges of reward hacking and extreme multi-objective optimization (i.e., trade-off of multiple and/or s…

HybridFlow: A Flexible and Efficient RLHF Framework

Authors: Guangming Sheng, Chi Zhang, Zilingfeng Ye et al. Published: 2024-09-28 arXiv: 2409.19256v2 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) is widely used in Large Language Model (LLM) alignment. Traditional RL can be modeled as a dataflow, where each node represents computation of a neural network (NN) and each edge denotes data dependencies between the NNs. RLHF complicates the dataflo…

MathDSL: A Domain-Specific Language for Concise Mathematical Solutions Via Program Synthesis

Authors: Sagnik Anupam, Maddy Bowers, Omar Costilla-Reyes et al. Published: 2024-09-26 arXiv: 2409.17490v3 PDF: Download

Abstract: We present MathDSL, a Domain-Specific Language (DSL) for mathematical equation solving, which, when deployed in program synthesis models, outperforms state-of-the-art reinforcement-learning-based methods. We also introduce a quantitative metric for measuring the conciseness of a mathematical solutio…

Reward-Robust RLHF in LLMs

Authors: Yuzi Yan, Xingzhou Lou, Jialian Li et al. Published: 2024-09-18 arXiv: 2409.15360v3 PDF: Download

Abstract: As Large Language Models (LLMs) continue to progress toward more advanced forms of intelligence, Reinforcement Learning from Human Feedback (RLHF) is increasingly seen as a key pathway toward achieving Artificial General Intelligence (AGI). However, the reliance on reward-model-based (RM-based) alig…

Tidal MerzA: Combining affective modelling and autonomous code generation through Reinforcement Learning

Authors: Elizabeth Wilson, György Fazekas, Geraint Wiggins Published: 2024-09-12 arXiv: 2409.07918v1 PDF: Download

Abstract: This paper presents Tidal-MerzA, a novel system designed for collaborative performances between humans and a machine agent in the context of live coding, specifically focusing on the generation of musical patterns. Tidal-MerzA fuses two foundational models: ALCAA (Affective Live Coding Autonomous Ag…

Policy Filtration for RLHF to Mitigate Noise in Reward Models

Authors: Chuheng Zhang, Wei Shen, Li Zhao et al. Published: 2024-09-11 arXiv: 2409.06957v5 PDF: Download

Abstract: While direct policy optimization methods exist, pioneering LLMs are fine-tuned with reinforcement learning from human feedback (RLHF) to generate better responses under the supervision of a reward model learned from preference data. One major challenge of RLHF is the inaccuracy of the intermediate r…

Large Language Model for Verilog Generation with Code-Structure-Guided Reinforcement Learning

Authors: Ning Wang, Bingkun Yao, Jie Zhou et al. Published: 2024-07-21 arXiv: 2407.18271v4 PDF: Download

Abstract: Recent advancements in large language models (LLMs) have sparked significant interest in the automatic generation of Register Transfer Level (RTL) designs, particularly using Verilog. Current research on this topic primarily focuses on pre-training and instruction tuning, but the effectiveness of th…

Multi-Step Reasoning with Large Language Models, a Survey

Authors: Aske Plaat, Annie Wong, Suzan Verberne et al. Published: 2024-07-16 arXiv: 2407.11511v3 PDF: Download

Abstract: Large language models (LLMs) with billions of parameters exhibit in-context learning abilities, enabling few-shot learning on tasks that the model was not specifically trained for. Traditional models achieve breakthrough performance on language tasks, but do not perform well on basic reasoning bench…

Reinforcement Learning from Human Feedback: Whose Culture, Whose Values, Whose Perspectives?

Authors: Kristian González Barman, Simon Lohse, Henk de Regt Published: 2024-07-02 arXiv: 2407.17482v2 PDF: Download

Abstract: We argue for the epistemic and ethical advantages of pluralism in Reinforcement Learning from Human Feedback (RLHF) in the context of Large Language Models (LLM). Drawing on social epistemology and pluralist philosophy of science, we suggest ways in which RHLF can be made more responsive to human ne…

Exploring Advanced Large Language Models with LLMsuite

Authors: Giorgio Roffo Published: 2024-07-01 arXiv: 2407.12036v2 PDF: Download

Abstract: This tutorial explores the advancements and challenges in the development of Large Language Models (LLMs) such as ChatGPT and Gemini. It addresses inherent limitations like temporal knowledge cutoffs, mathematical inaccuracies, and the generation of incorrect information, proposing solutions like Re…

Mental Modeling of Reinforcement Learning Agents by Language Models

Authors: Wenhao Lu, Xufeng Zhao, Josua Spisak et al. Published: 2024-06-26 arXiv: 2406.18505v1 PDF: Download

Abstract: Can emergent language models faithfully model the intelligence of decision-making agents? Though modern language models exhibit already some reasoning ability, and theoretically can potentially express any probable distribution over tokens, it remains underexplored how the world knowledge these pret…

Authors: Ju-Seung Byun, Jiyun Chun, Jihyung Kil et al. Published: 2024-06-25 arXiv: 2407.00087v2 PDF: Download

Abstract: Large Multimodal Models (LMMs) excel at comprehending human instructions and demonstrate remarkable results across a broad spectrum of tasks. Reinforcement Learning from Human Feedback (RLHF) and AI Feedback (RLAIF) further refine LLMs by aligning them with specific preferences. These methods primar…

Measuring memorization in RLHF for code completion

Authors: Aneesh Pappu, Billy Porter, Ilia Shumailov et al. Published: 2024-06-17 arXiv: 2406.11715v2 PDF: Download

Abstract: Reinforcement learning with human feedback (RLHF) has become the dominant method to align large models to user preferences. Unlike fine-tuning, for which there are many studies regarding training data memorization, it is not clear how memorization is affected by or introduced in the RLHF alignment p…

Online Bandit Learning with Offline Preference Data for Improved RLHF

Authors: Akhil Agnihotri, Rahul Jain, Deepak Ramachandran et al. Published: 2024-06-13 arXiv: 2406.09574v4 PDF: Download

Abstract: Reinforcement Learning with Human Feedback (RLHF) is at the core of fine-tuning methods for generative AI models for language and images. Such feedback is often sought as rank or preference feedback from human raters, as opposed to eliciting scores since the latter tends to be noisy. On the other ha…

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Authors: Tengyang Xie, Dylan J. Foster, Akshay Krishnamurthy et al. Published: 2024-05-31 arXiv: 2405.21046v1 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) has emerged as a central tool for language model alignment. We consider online exploration in RLHF, which exploits interactive access to human or AI feedback by deliberately encouraging the model to produce diverse, maximally informative responses. B…

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF

Authors: Shicong Cen, Jincheng Mei, Katayoon Goshvadi et al. Published: 2024-05-29 arXiv: 2405.19320v4 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) has demonstrated great promise in aligning large language models (LLMs) with human preference. Depending on the availability of preference data, both online and offline RLHF are active areas of investigation. A key bottleneck is understanding how to …

No $D_{\text{train}}$: Model-Agnostic Counterfactual Explanations Using Reinforcement Learning

Authors: Xiangyu Sun, Raquel Aoki, Kevin H. Wilson Published: 2024-05-28 arXiv: 2405.18563v2 PDF: Download

Abstract: Machine learning (ML) methods have experienced significant growth in the past decade, yet their practical application in high-impact real-world domains has been hindered by their opacity. When ML methods are responsible for making critical decisions, stakeholders often require insights into how to a…

A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learning

Authors: Abdulaziz Almuzairee, Nicklas Hansen, Henrik I. Christensen Published: 2024-05-27 arXiv: 2405.17416v2 PDF: Download

Abstract: Q-learning algorithms are appealing for real-world applications due to their data-efficiency, but they are very prone to overfitting and training instabilities when trained from visual observations. Prior work, namely SVEA, finds that selective application of data augmentation can improve the visual…

Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search

Authors: Nicola Dainese, Matteo Merler, Minttu Alakuijala et al. Published: 2024-05-24 arXiv: 2405.15383v2 PDF: Download

Abstract: In this work we consider Code World Models, world models generated by a Large Language Model (LLM) in the form of Python code for model-based Reinforcement Learning (RL). Calling code instead of LLMs for planning has potential to be more precise, reliable, interpretable, and extremely efficient. How…

Knowledge Graph Reasoning with Self-supervised Reinforcement Learning

Authors: Ying Ma, Owen Burns, Mingqiu Wang et al. Published: 2024-05-22 arXiv: 2405.13640v2 PDF: Download

Abstract: Reinforcement learning (RL) is an effective method of finding reasoning pathways in incomplete knowledge graphs (KGs). To overcome the challenges of a large action space, a self-supervised pre-training method is proposed to warm up the policy network before the RL training stage. To alleviate the di…

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

Authors: Jian Hu, Xibin Wu, Wei Shen et al. Published: 2024-05-20 arXiv: 2405.11143v6 PDF: Download

Abstract: Large Language Models (LLMs) fine-tuned via Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) significantly improve the alignment of human-AI values, further raising the upper bound of AI capabilities, particularly in reasoning-intensive, lon…

RLHF Workflow: From Reward Modeling to Online RLHF

Authors: Hanze Dong, Wei Xiong, Bo Pang et al. Published: 2024-05-13 arXiv: 2405.07863v3 PDF: Download

Abstract: We present the workflow of Online Iterative Reinforcement Learning from Human Feedback (RLHF) in this technical report, which is widely reported to outperform its offline counterpart by a large margin in the recent large language model (LLM) literature. However, existing open-source RLHF projects ar…

DPO Meets PPO: Reinforced Token Optimization for RLHF

Authors: Han Zhong, Zikang Shan, Guhao Feng et al. Published: 2024-04-29 arXiv: 2404.18922v4 PDF: Download

Abstract: In the classical Reinforcement Learning from Human Feedback (RLHF) framework, Proximal Policy Optimization (PPO) is employed to learn from sparse, sentence-level rewards — a challenging scenario in traditional deep reinforcement learning. Despite the great successes of PPO in the alignment of large…

Reinforcement Learning with Generative Models for Compact Support Sets

Authors: Nico Schiavone, Xingyu Li Published: 2024-04-25 arXiv: 2404.16300v1 PDF: Download

Abstract: Foundation models contain a wealth of information from their vast number of training samples. However, most prior arts fail to extract this information in a precise and efficient way for small sample sizes. In this work, we propose a framework utilizing reinforcement learning as a control for founda…

CompilerDream: Learning a Compiler World Model for General Code Optimization

Authors: Chaoyi Deng, Jialong Wu, Ningya Feng et al. Published: 2024-04-24 arXiv: 2404.16077v3 PDF: Download

Abstract: Effective code optimization in compilers is crucial for computer and software engineering. The success of these optimizations primarily depends on the selection and ordering of the optimization passes applied to the code. While most compilers rely on a fixed sequence of optimization passes, current …

Authors: Jessica Dai, Eve Fleisig Published: 2024-04-19 arXiv: 2404.13038v1 PDF: Download

Abstract: Recent work on the limitations of using reinforcement learning from human feedback (RLHF) to incorporate human preferences into model behavior often raises social choice theory as a reference point. Social choice theory’s analysis of settings such as voting mechanisms provides technical infrastructu…

RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs

Authors: Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari et al. Published: 2024-04-12 arXiv: 2404.08555v2 PDF: Download

Abstract: State-of-the-art large language models (LLMs) have become indispensable tools for various tasks. However, training LLMs to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedbac…

Improving Language Model Reasoning with Self-motivated Learning

Authors: Yunlong Feng, Yang Xu, Libo Qin et al. Published: 2024-04-10 arXiv: 2404.07017v3 PDF: Download

Abstract: Large-scale high-quality training data is important for improving the performance of models. After trained with data that has rationales (reasoning steps), models gain reasoning capability. However, the dataset with high-quality rationales is relatively scarce due to the high annotation cost. To add…

Dreaming of Many Worlds: Learning Contextual World Models Aids Zero-Shot Generalization

Authors: Sai Prasanna, Karim Farid, Raghu Rajan et al. Published: 2024-03-16 arXiv: 2403.10967v2 PDF: Download

Abstract: Zero-shot generalization (ZSG) to unseen dynamics is a major challenge for creating generally capable embodied agents. To address the broader challenge, we start with the simpler setting of contextual reinforcement learning (cRL), assuming observability of the context values that parameterize the va…

Parameter Efficient Reinforcement Learning from Human Feedback

Authors: Hakim Sidahmed, Samrat Phatale, Alex Hutcheson et al. Published: 2024-03-15 arXiv: 2403.10704v2 PDF: Download

Abstract: While Reinforcement Learning from Human Feedback (RLHF) effectively aligns pretrained Large Language and Vision-Language Models (LLMs, and VLMs) with human preferences, its computational cost and complexity hamper its wider adoption. To alleviate some of the computational burden of fine-tuning, para…

Provable Multi-Party Reinforcement Learning with Diverse Human Feedback

Authors: Huiying Zhong, Zhun Deng, Weijie J. Su et al. Published: 2024-03-08 arXiv: 2403.05006v1 PDF: Download

Abstract: Reinforcement learning with human feedback (RLHF) is an emerging paradigm to align models with human preferences. Typically, RLHF aggregates preferences from multiple individuals who have diverse viewpoints that may conflict with each other. Our work \textit{initiates} the theoretical study of multi…

Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy

Authors: Yu Zhu, Chuxiong Sun, Wenfei Yang et al. Published: 2024-03-07 arXiv: 2403.04283v1 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) is the prevailing approach to ensure Large Language Models (LLMs) align with human values. However, existing RLHF methods require a high computational cost, one main reason being that RLHF assigns both the generation and alignment tasks to the LLM si…

Teaching Large Language Models to Reason with Reinforcement Learning

Authors: Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy et al. Published: 2024-03-07 arXiv: 2403.04642v1 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (\textbf{RLHF}) has emerged as a dominant approach for aligning LLM outputs with human preferences. Inspired by the success of RLHF, we study the performance of multiple algorithms that learn from feedback (Expert Iteration, Proximal Policy Optimization (\t…

Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization

Authors: Yihan Du, Anna Winnicki, Gal Dalal et al. Published: 2024-02-15 arXiv: 2402.10342v2 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) has achieved impressive empirical successes while relying on a small amount of human feedback. However, there is limited theoretical justification for this phenomenon. Additionally, most recent studies focus on value-based algorithms despite the rece…

Online Iterative Reinforcement Learning from Human Feedback with General Preference Model

Authors: Chenlu Ye, Wei Xiong, Yuheng Zhang et al. Published: 2024-02-11 arXiv: 2402.07314v3 PDF: Download

Abstract: We investigate Reinforcement Learning from Human Feedback (RLHF) in the context of a general preference oracle. In particular, we do not assume the existence of a reward function and an oracle preference signal drawn from the Bradley-Terry model as most of the prior works do. We consider a standard …

Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF

Authors: Han Shen, Zhuoran Yang, Tianyi Chen Published: 2024-02-10 arXiv: 2402.06886v3 PDF: Download

Abstract: Bilevel optimization has been recently applied to many machine learning tasks. However, their applications have been restricted to the supervised learning setting, where static objective functions with benign structures are considered. But bilevel problems such as incentive design, inverse reinforce…

Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning

Authors: Zhiheng Xi, Wenxiang Chen, Boyang Hong et al. Published: 2024-02-08 arXiv: 2402.05808v2 PDF: Download

Abstract: In this paper, we propose R$^3$: Learning Reasoning through Reverse Curriculum Reinforcement Learning (RL), a novel method that employs only outcome supervision to achieve the benefits of process supervision for large language models. The core challenge in applying RL to complex reasoning is to iden…

Code as Reward: Empowering Reinforcement Learning with VLMs

Authors: David Venuto, Sami Nur Islam, Martin Klissarov et al. Published: 2024-02-07 arXiv: 2402.04764v1 PDF: Download

Abstract: Pre-trained Vision-Language Models (VLMs) are able to understand visual concepts, describe and decompose complex tasks into sub-tasks, and provide feedback on task completion. In this paper, we aim to leverage these capabilities to support the training of reinforcement learning (RL) agents. In princ…

A Theoretical Framework for Partially Observed Reward-States in RLHF

Authors: Chinmaya Kausik, Mirco Mutti, Aldo Pacchiano et al. Published: 2024-02-05 arXiv: 2402.03282v3 PDF: Download

Abstract: The growing deployment of reinforcement learning from human feedback (RLHF) calls for a deeper theoretical investigation of its underlying models. The prevalent models of RLHF do not account for neuroscience-backed, partially-observed “internal states” that can affect human feedback, nor do they acc…

Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback

Authors: Yifu Yuan, Jianye Hao, Yi Ma et al. Published: 2024-02-04 arXiv: 2402.02423v2 PDF: Download

Abstract: Reinforcement Learning with Human Feedback (RLHF) has received significant attention for performing tasks without the need for costly manual reward design by aligning human preferences. It is crucial to consider diverse human feedback types and various learning methods in different environments. How…

StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback

Authors: Shihan Dou, Yan Liu, Haoxiang Jia et al. Published: 2024-02-02 arXiv: 2402.01391v2 PDF: Download

Abstract: The advancement of large language models (LLMs) has significantly propelled the field of code generation. Previous work integrated reinforcement learning (RL) with compiler feedback for exploring the output space of LLMs to enhance code generation quality. However, the lengthy code generated by LLMs…

2023

A Survey of Reinforcement Learning from Human Feedback

Authors: Timo Kaufmann, Paul Weng, Viktor Bengs et al. Published: 2023-12-22 arXiv: 2312.14925v2 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) is a variant of reinforcement learning (RL) that learns from human feedback instead of relying on an engineered reward function. Building on prior work on the related setting of preference-based reinforcement learning (PbRL), it stands at the interse…

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training

Authors: Youshao Xiao, Zhenglei Zhou, Fagui Mao et al. Published: 2023-12-19 arXiv: 2312.11819v3 PDF: Download

Abstract: Recently, ChatGPT or InstructGPT like large language models (LLM) has made a significant impact in the AI world. Many works have attempted to reproduce the complex InstructGPT’s training pipeline, namely Reinforcement Learning with Human Feedback (RLHF). However, the mainstream distributed RLHF trai…

Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

Authors: Wei Xiong, Hanze Dong, Chenlu Ye et al. Published: 2023-12-18 arXiv: 2312.11456v4 PDF: Download

Abstract: This paper studies the alignment process of generative models with Reinforcement Learning from Human Feedback (RLHF). We first identify the primary challenges of existing popular methods like offline PPO and offline DPO as lacking in strategical exploration of the environment. Then, to understand th…

Improving Environment Robustness of Deep Reinforcement Learning Approaches for Autonomous Racing Using Bayesian Optimization-based Curriculum Learning

Authors: Rohan Banerjee, Prishita Ray, Mark Campbell Published: 2023-12-16 arXiv: 2312.10557v1 PDF: Download

Abstract: Deep reinforcement learning (RL) approaches have been broadly applied to a large number of robotics tasks, such as robot manipulation and autonomous driving. However, an open problem in deep RL is learning policies that are robust to variations in the environment, which is an important condition for…

Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF

Authors: Anand Siththaranjan, Cassidy Laidlaw, Dylan Hadfield-Menell Published: 2023-12-13 arXiv: 2312.08358v2 PDF: Download

Abstract: In practice, preference learning from human feedback depends on incomplete data with hidden context. Hidden context refers to data that affects the feedback received, but which is not represented in the data used to train a preference model. This captures common issues of data collection, such as ha…

RLHF and IIA: Perverse Incentives

Authors: Wanqiao Xu, Shi Dong, Xiuyuan Lu et al. Published: 2023-12-02 arXiv: 2312.01057v3 PDF: Download

Abstract: Existing algorithms for reinforcement learning from human feedback (RLHF) can incentivize responses at odds with preferences because they are based on models that assume independence of irrelevant alternatives (IIA). The perverse incentives induced by IIA hinder innovations on query formats and lear…

Reinforcement Learning from Statistical Feedback: the Journey from AB Testing to ANT Testing

Authors: Feiyang Han, Yimin Wei, Zhaofeng Liu et al. Published: 2023-11-24 arXiv: 2311.14766v1 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) has played a crucial role in the success of large models such as ChatGPT. RLHF is a reinforcement learning framework which combines human feedback to improve learning effectiveness and performance. However, obtaining preferences feedback manually is …

An introduction to reinforcement learning for neuroscience

Authors: Kristopher T. Jensen Published: 2023-11-13 arXiv: 2311.07315v3 PDF: Download

Abstract: Reinforcement learning (RL) has a rich history in neuroscience, from early work on dopamine as a reward prediction error signal (Schultz et al., 1997) to recent work proposing that the brain could implement a form of ‘distributional reinforcement learning’ popularized in machine learning (Dabney et …

Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis

Authors: Philip John Gorinski, Matthieu Zimmer, Gerasimos Lampouras et al. Published: 2023-10-20 arXiv: 2310.13669v1 PDF: Download

Abstract: The advent of large pre-trained language models in the domain of Code Synthesis has shown remarkable performance on various benchmarks, treating the problem of Code Generation in a fashion similar to Natural Language Generation, trained with a Language Modelling (LM) objective. In addition, the prop…

Safe RLHF: Safe Reinforcement Learning from Human Feedback

Authors: Josef Dai, Xuehai Pan, Ruiyang Sun et al. Published: 2023-10-19 arXiv: 2310.12773v1 PDF: Download

Abstract: With the development of large language models (LLMs), striking a balance between the performance and safety of AI systems has never been more critical. However, the inherent tension between the objectives of helpfulness and harmlessness presents a significant challenge during LLM training. To addres…

Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning

Authors: Juan Rocamonde, Victoriano Montesinos, Elvis Nava et al. Published: 2023-10-19 arXiv: 2310.12921v2 PDF: Download

Abstract: Reinforcement learning (RL) requires either manually specifying a reward function, which is often infeasible, or learning a reward model from a large amount of human feedback, which is often very expensive. We study a more sample-efficient alternative: using pretrained vision-language models (VLMs) …

Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond

Authors: Hao Sun Published: 2023-10-09 arXiv: 2310.06147v1 PDF: Download

Abstract: Recent advancements in Large Language Models (LLMs) have garnered wide attention and led to successful products such as ChatGPT and GPT-4. Their proficiency in adhering to instructions and delivering harmless, helpful, and honest (3H) responses can largely be attributed to the technique of Reinforce…

A Long Way to Go: Investigating Length Correlations in RLHF

Authors: Prasann Singhal, Tanya Goyal, Jiacheng Xu et al. Published: 2023-10-05 arXiv: 2310.03716v2 PDF: Download

Abstract: Great success has been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation, particularly for “helpfulness” in tasks like dialogue and web question answering. Alongside these improvements, howeve…

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Authors: Harrison Lee, Samrat Phatale, Hassan Mansoor et al. Published: 2023-09-01 arXiv: 2309.00267v3 PDF: Download

Abstract: Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al., offers a promising alternative that trains the rew…

WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct

Authors: Haipeng Luo, Qingfeng Sun, Can Xu et al. Published: 2023-08-18 arXiv: 2308.09583v3 PDF: Download

Abstract: Large language models (LLMs), such as GPT-4, have shown remarkable performance in natural language processing (NLP) tasks, including challenging mathematical reasoning. However, most existing open-source models are only pre-trained on large-scale internet data and without math-related optimization. …

RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback

Authors: Yannick Metz, David Lindner, Raphaël Baur et al. Published: 2023-08-08 arXiv: 2308.04332v1 PDF: Download

Abstract: To use reinforcement learning from human feedback (RLHF) in practical applications, it is crucial to learn reward models from diverse sources of human feedback and to consider human factors involved in providing feedback of different types. However, the systematic study of learning from diverse type…

Reinforcement Learning and Data-Generation for Syntax-Guided Synthesis

Authors: Julian Parsert, Elizabeth Polgreen Published: 2023-07-13 arXiv: 2307.09564v2 PDF: Download

Abstract: Program synthesis is the task of automatically generating code based on a specification. In Syntax-Guided Synthesis (SyGuS) this specification is a combination of a syntactic template and a logical formula, and the result is guaranteed to satisfy both. We present a reinforcement-learning guided al…

RLTF: Reinforcement Learning from Unit Test Feedback

Authors: Jiate Liu, Yiqin Zhu, Kaiwen Xiao et al. Published: 2023-07-10 arXiv: 2307.04349v2 PDF: Download

Abstract: The goal of program synthesis, or code generation, is to generate executable code based on given descriptions. Recently, there has been an increasing number of studies employing reinforcement learning (RL) to improve the performance of large language models (LLMs) for code. However, current represen…

Can Differentiable Decision Trees Enable Interpretable Reward Learning from Human Feedback?

Authors: Akansha Kalra, Daniel S. Brown Published: 2023-06-22 arXiv: 2306.13004v6 PDF: Download

Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular paradigm for capturing human intent to alleviate the challenges of hand-crafting the reward values. Despite the increasing interest in RLHF, most works learn black box reward functions that while expressive are difficult to i…

Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models’ Reasoning Performance

Authors: Yao Fu, Litu Ou, Mingyu Chen et al. Published: 2023-05-26 arXiv: 2305.17306v1 PDF: Download

Abstract: As large language models (LLMs) are continuously being developed, their evaluation becomes increasingly important yet challenging. This work proposes Chain-of-Thought Hub, an open-source evaluation suite on the multi-step reasoning capabilities of large language models. We are interested in this set…

Coarse-Tuning Models of Code with Reinforcement Learning Feedback

Authors: Abhinav Jain, Chima Adiole, Swarat Chaudhuri et al. Published: 2023-05-25 arXiv: 2305.18341v2 PDF: Download

Abstract: Large Language Models (LLMs) pre-trained on code have recently emerged as the dominant approach to program synthesis. However, these models are trained using next-token prediction, which ignores the syntax and semantics of code. We propose RLCF, that further trains a pre-trained LLM via reinforcemen…

Language Model Self-improvement by Reinforcement Learning Contemplation

Authors: Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li et al. Published: 2023-05-23 arXiv: 2305.14483v1 PDF: Download

Abstract: Large Language Models (LLMs) have exhibited remarkable performance across various natural language processing (NLP) tasks. However, fine-tuning these models often necessitates substantial supervision, which can be expensive and time-consuming to obtain. This paper introduces a novel unsupervised met…

Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actions

Authors: Lina Mezghani, Piotr Bojanowski, Karteek Alahari et al. Published: 2023-04-18 arXiv: 2304.11063v1 PDF: Download

Abstract: The success of transformer models trained with a language modeling objective brings a promising opportunity to the reinforcement learning framework. Decision Transformer is a step towards this direction, showing how to train transformers with a similar next-step prediction objective on offline data…

Improved Tree Search for Automatic Program Synthesis

Authors: Aran Carmon, Lior Wolf Published: 2023-03-13 arXiv: 2303.07166v1 PDF: Download

Abstract: In the task of automatic program synthesis, one obtains pairs of matching inputs and outputs and generates a computer program, in a particular domain-specific language (DSL), which given each sample input returns the matching output. A key element is being able to perform an efficient search in the …

Collaborating with language models for embodied reasoning

Authors: Ishita Dasgupta, Christine Kaeser-Chen, Kenneth Marino et al. Published: 2023-02-01 arXiv: 2302.00763v1 PDF: Download

Abstract: Reasoning in a complex and ambiguous environment is a key goal for Reinforcement Learning (RL) agents. While some sophisticated RL agents can successfully solve difficult tasks, they require a large amount of training data and often struggle to generalize to new unseen environments and new tasks. On…

Execution-based Code Generation using Deep Reinforcement Learning

Authors: Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni et al. Published: 2023-01-31 arXiv: 2301.13816v4 PDF: Download

Abstract: The utilization of programming language (PL) models, pre-trained on large-scale code corpora, as a means of automating software engineering processes has demonstrated considerable potential in streamlining various code generation tasks such as code completion, code translation, and program synthesis…

Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons

Authors: Banghua Zhu, Jiantao Jiao, Michael I. Jordan Published: 2023-01-26 arXiv: 2301.11270v5 PDF: Download

Abstract: We provide a theoretical framework for Reinforcement Learning with Human Feedback (RLHF). Our analysis shows that when the true reward function is linear, the widely used maximum likelihood estimator (MLE) converges under both the Bradley-Terry-Luce (BTL) model and the Plackett-Luce (PL) model. Howe…

Time-Myopic Go-Explore: Learning A State Representation for the Go-Explore Paradigm

Authors: Marc Höftmann, Jan Robine, Stefan Harmeling Published: 2023-01-13 arXiv: 2301.05635v1 PDF: Download

Abstract: Very large state spaces with a sparse reward signal are difficult to explore. The lack of a sophisticated guidance results in a poor performance for numerous reinforcement learning algorithms. In these cases, the commonly used random exploration is often not helpful. The literature shows that this k…

2022

Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics

Authors: Krishan Rana, Ming Xu, Brendan Tidd et al. Published: 2022-11-04 arXiv: 2211.02231v1 PDF: Download

Abstract: Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. Skills are typically extracted from expert demonstrations and are embedded into a latent space from which they can be sampled as actions by a high-level RL agent. H…

ReAct: Synergizing Reasoning and Acting in Language Models

Authors: Shunyu Yao, Jeffrey Zhao, Dian Yu et al. Published: 2022-10-06 arXiv: 2210.03629v3 PDF: Download

Abstract: While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics…

CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

Authors: Hung Le, Yue Wang, Akhilesh Deepak Gotmare et al. Published: 2022-07-05 arXiv: 2207.01780v3 PDF: Download

Abstract: Program synthesis or code generation aims to generate a program that satisfies a problem specification. Recent approaches using large-scale pretrained language models (LMs) have shown promising results, yet they have some critical limitations. In particular, they often follow a standard supervised f…

GALOIS: Boosting Deep Reinforcement Learning via Generalizable Logic Synthesis

Authors: Yushi Cao, Zhiming Li, Tianpei Yang et al. Published: 2022-05-27 arXiv: 2205.13728v1 PDF: Download

Abstract: Despite achieving superior performance in human-level control problems, unlike humans, deep reinforcement learning (DRL) lacks high-order intelligence (e.g., logic deduction and reuse), thus it behaves ineffectively than humans regarding learning and generalization in complex problems. Previous work…

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Authors: Yuntao Bai, Andy Jones, Kamal Ndousse et al. Published: 2022-04-12 arXiv: 2204.05862v1 PDF: Download

Abstract: We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills s…

Source: reference/papers/LLM-Code-Generation/PAPERS.md on GitHub.