Last Updated: 2025-11-12
Total Papers: 111
2025
Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference
Authors: Matteo Cercola, Valeria Capretti, Simone Formentin Published: 2025-11-06 arXiv: 2511.04286v1 PDF: Download
Abstract: Learning from human preferences is a cornerstone of aligning machine learning models with subjective human judgments. Yet, collecting such preference data is often costly and time-consuming, motivating the need for more efficient learning paradigms. Two established approaches offer complementary adv…
Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
Authors: Keertana Chidambaram, Karthik Vinary Seetharaman, Vasilis Syrgkanis Published: 2025-10-17 arXiv: 2510.15716v1 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) has become central to aligning large language models with human values, typically by first learning a reward model from preference data which is then used to update the model with reinforcement learning. Recent alternatives such as Direct Preference …
Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization
Authors: Jason Bohne, Pawel Polak, David Rosenberg et al. Published: 2025-10-09 arXiv: 2510.08256v1 PDF: Download
Abstract: Direct Preference Optimization (DPO) has recently emerged as a simple and effective alternative to reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs) with user preferences. However, existing DPO formulations rely on a single monolithic model, which limits the…
Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment
Authors: Ziyi Chen, Junyi Li, Peiran Yu et al. Published: 2025-10-07 arXiv: 2510.05526v1 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) are important techniques to align large language models (LLM) with human preference. However, the quality of RLHF and DPO training is seriously compromised by \textit{\textbf{C}orrupted} preference, reward \te…
Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models through Reinforcement Learning from Ranking Feedback
Authors: Derek Shi, Ruben Glatt, Christine Klymko et al. Published: 2025-10-02 arXiv: 2510.02561v1 PDF: Download
Abstract: Recent advances in large video-language models (VLMs) rely on extensive fine-tuning techniques that strengthen alignment between textual and visual comprehension. Leading pipelines typically pair supervised fine-tuning (SFT) with reinforcement learning from preference data to enhance video comprehen…
Adaptive Margin RLHF via Preference over Preferences
Authors: Yaswanth Chittepu, Prasann Singhal, Greg Durrett et al. Published: 2025-09-26 arXiv: 2509.22851v2 PDF: Download
Abstract: Margin-based optimization is fundamental to improving generalization and robustness in classification tasks. In the context of reward model learning from preferences within Reinforcement Learning from Human Feedback (RLHF), existing methods typically rely on no margins, fixed margins, or margins tha…
Improving LLM Safety and Helpfulness using SFT and DPO: A Study on OPT-350M
Authors: Piyush Pant Published: 2025-09-10 arXiv: 2509.09055v1 PDF: Download
Abstract: This research investigates the effectiveness of alignment techniques, Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and a combined SFT+DPO approach on improving the safety and helpfulness of the OPT-350M language model. Utilizing the Anthropic Helpful-Harmless RLHF dataset, we …
Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Authors: Xuan Qi, Rongwu Xu, Zhijing Jin Published: 2025-08-06 arXiv: 2508.04149v1 PDF: Download
Abstract: Aligning large language models (LLMs) with human preferences is a critical challenge in AI research. While methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are widely used, they often rely on large, costly preference datasets. The current work l…
Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback
Authors: Johannes Ackermann, Takashi Ishida, Masashi Sugiyama Published: 2025-07-21 arXiv: 2507.15507v1 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) allows us to train models, such as language models (LMs), to follow complex human preferences. In RLHF for LMs, we first train an LM using supervised fine-tuning, sample pairs of responses, obtain human feedback, and use the resulting data to train a…
Aligning Humans and Robots via Reinforcement Learning from Implicit Human Feedback
Authors: Suzie Kim, Hye-Bin Shin, Seong-Whan Lee Published: 2025-07-17 arXiv: 2507.13171v1 PDF: Download
Abstract: Conventional reinforcement learning (RL) ap proaches often struggle to learn effective policies under sparse reward conditions, necessitating the manual design of complex, task-specific reward functions. To address this limitation, rein forcement learning from human feedback (RLHF) has emerged as a …
Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback
Authors: Jan Kompatscher, Danqing Shi, Giovanna Varni et al. Published: 2025-07-06 arXiv: 2507.04340v1 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has emerged as a key enabling technology for aligning AI behavior with human preferences. The traditional way to collect data in RLHF is via pairwise comparisons: human raters are asked to indicate which one of two samples they prefer. We present an …
Multi-Preference Lambda-weighted Listwise DPO for Small-Scale Model Alignment
Authors: Yuhui Sun, Xiyao Wang, Zixi Li et al. Published: 2025-06-24 arXiv: 2506.19780v5 PDF: Download
Abstract: Large language models (LLMs) demonstrate strong generalization across a wide range of language tasks, but often generate outputs that misalign with human preferences. Reinforcement Learning from Human Feedback (RLHF) addresses this by optimizing models toward human preferences using a learned reward…
Mapping Neural Signals to Agent Performance, A Step Towards Reinforcement Learning from Neural Feedback
Authors: Julia Santaniello, Matthew Russell, Benson Jiang et al. Published: 2025-06-14 arXiv: 2506.12636v1 PDF: Download
Abstract: Implicit Human-in-the-Loop Reinforcement Learning (HITL-RL) is a methodology that integrates passive human feedback into autonomous agent training while minimizing human workload. However, existing methods often rely on active instruction, requiring participants to teach an agent through unnatural e…
Omni-DPO: A Dual-Perspective Paradigm for Dynamic Preference Learning of LLMs
Authors: Shangpin Peng, Weinong Wang, Zhuotao Tian et al. Published: 2025-06-11 arXiv: 2506.10054v2 PDF: Download
Abstract: Direct Preference Optimization (DPO) has become a cornerstone of reinforcement learning from human feedback (RLHF) due to its simplicity and efficiency. However, existing DPO-based approaches typically treat all preference pairs uniformly, ignoring critical variations in their inherent quality and l…
Multi-Task Reward Learning from Human Ratings
Authors: Mingkang Wu, Devin White, Evelyn Rose et al. Published: 2025-06-10 arXiv: 2506.09183v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has become a key factor in aligning model behavior with users’ goals. However, while humans integrate multiple strategies when making decisions, current RLHF approaches often simplify this process by modeling human reasoning through isolated tasks su…
AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization
Authors: Zixuan Jiang, Renjing Xu Published: 2025-06-08 arXiv: 2506.07035v1 PDF: Download
Abstract: Deciphering protein function remains a fundamental challenge in protein representation learning. The task presents significant difficulties for protein language models (PLMs) due to the sheer volume of functional annotation categories and the highly imbalanced distribution of annotated instances acr…
Provable Reinforcement Learning from Human Feedback with an Unknown Link Function
Authors: Qining Zhang, Lei Ying Published: 2025-06-03 arXiv: 2506.03066v1 PDF: Download
Abstract: Link functions, which characterize how human preferences are generated from the value function of an RL problem, are a crucial component in designing RLHF algorithms. Almost all RLHF algorithms, including state-of-the-art ones in empirical studies such as DPO and PPO, assume the link function is kno…
Alignment as Distribution Learning: Your Preference Model is Explicitly a Language Model
Authors: Jihun Yun, Juno Kim, Jongho Park et al. Published: 2025-06-02 arXiv: 2506.01523v1 PDF: Download
Abstract: Alignment via reinforcement learning from human feedback (RLHF) has become the dominant paradigm for controlling the quality of outputs from large language models (LLMs). However, when viewed as `loss + regularization,’ the standard RLHF objective lacks theoretical justification and incentivizes deg…
Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data
Authors: Christopher Lee Lübbers Published: 2025-05-28 arXiv: 2506.02018v1 PDF: Download
Abstract: Paraphrasing re-expresses meaning to enhance applications like text simplification, machine translation, and question-answering. Specific paraphrase types facilitate accurate semantic analysis and robust language models. However, existing paraphrase-type generation methods often misalign with human …
A Unified Theoretical Analysis of Private and Robust Offline Alignment: from RLHF to DPO
Authors: Xingyu Zhou, Yulian Wu, Francesco Orabona Published: 2025-05-21 arXiv: 2505.15694v1 PDF: Download
Abstract: In this paper, we theoretically investigate the effects of noisy labels in offline alignment, with a focus on the interplay between privacy and robustness against adversarial corruption. Specifically, under linear modeling assumptions, we present a unified analysis covering both reinforcement learni…
Contextual Online Uncertainty-Aware Preference Learning for Human Feedback
Authors: Nan Lu, Ethan X. Fang, Junwei Lu Published: 2025-04-27 arXiv: 2504.19342v2 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) has become a pivotal paradigm in artificial intelligence to align large models with human preferences. In this paper, we propose a novel statistical framework to simultaneously conduct the online decision-making and statistical inference on the optim…
Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
Authors: Junshu Pan, Wei Shen, Shulin Huang et al. Published: 2025-04-22 arXiv: 2504.15843v2 PDF: Download
Abstract: Direct Preference Optimization (DPO) simplifies reinforcement learning from human feedback (RLHF) for large language models (LLMs) by directly optimizing human preferences without an explicit reward model. We find that during DPO training, the reference model plays the role of a data weight adjuster…
Reinforcement Learning from Multi-level and Episodic Human Feedback
Authors: Muhammad Qasim Elahi, Somtochukwu Oguchienti, Maheed H. Ahmed et al. Published: 2025-04-20 arXiv: 2504.14732v3 PDF: Download
Abstract: Designing an effective reward function has long been a challenge in reinforcement learning, particularly for complex tasks in unstructured environments. To address this, various learning paradigms have emerged that leverage different forms of human input to specify or refine the reward function. Rei…
Governance Challenges in Reinforcement Learning from Human Feedback: Evaluator Rationality and Reinforcement Stability
Authors: Dana Alsagheer, Abdulrahman Kamal, Mohammad Kamal et al. Published: 2025-04-17 arXiv: 2504.13972v1 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) is central in aligning large language models (LLMs) with human values and expectations. However, the process remains susceptible to governance challenges, including evaluator bias, inconsistency, and the unreliability of feedback. This study examines…
2D-Curri-DPO: Two-Dimensional Curriculum Learning for Direct Preference Optimization
Authors: Mengyang Li, Zhong Zhang Published: 2025-04-10 arXiv: 2504.07856v3 PDF: Download
Abstract: Aligning large language models with human preferences is crucial for their safe deployment. While Direct Preference Optimization (DPO) offers an efficient alternative to reinforcement learning from human feedback, traditional DPO methods are limited by their reliance on single preference pairs. Rece…
Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
Authors: Kai Ye, Hongyi Zhou, Jin Zhu et al. Published: 2025-04-03 arXiv: 2504.03784v4 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has emerged as a key technique for aligning the output of large language models (LLMs) with human preferences. To learn the reward function, most existing RLHF algorithms use the Bradley-Terry model, which relies on assumptions about human preference…
Sharpe Ratio-Guided Active Learning for Preference Optimization in RLHF
Authors: Syrine Belakaria, Joshua Kazdan, Charles Marx et al. Published: 2025-03-28 arXiv: 2503.22137v1 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has become a cornerstone of the training and alignment pipeline for large language models (LLMs). Recent advances, such as direct preference optimization (DPO), have simplified the preference learning step. However, collecting preference data remains…
Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
Authors: Wei Shen, Guanlin Liu, Zheng Wu et al. Published: 2025-03-28 arXiv: 2503.22230v3 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) is crucial for aligning large language models with human preferences. While recent research has focused on algorithmic improvements, the importance of prompt-data construction has been overlooked. This paper addresses this gap by exploring data-drive…
Policy Teaching via Data Poisoning in Learning from Human Preferences
Authors: Andi Nika, Jonathan Nöther, Debmalya Mandal et al. Published: 2025-03-13 arXiv: 2503.10228v1 PDF: Download
Abstract: We study data poisoning attacks in learning from human preferences. More specifically, we consider the problem of teaching/enforcing a target policy $π^\dagger$ by synthesizing preference data. We seek to understand the susceptibility of different preference-based learning paradigms to poisoned pref…
Strategyproof Reinforcement Learning from Human Feedback
Authors: Thomas Kleine Buening, Jiarui Gan, Debmalya Mandal et al. Published: 2025-03-12 arXiv: 2503.09561v2 PDF: Download
Abstract: We study Reinforcement Learning from Human Feedback (RLHF) in settings where multiple labelers may strategically misreport feedback to steer the learned policy toward their own preferences. We show that existing RLHF algorithms, including recent pluralistic methods, are not strategyproof, and that e…
Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback
Authors: Runlong Zhou, Maryam Fazel, Simon S. Du Published: 2025-03-11 arXiv: 2503.08942v3 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has become essential for improving language model capabilities, but traditional approaches rely on the assumption that human preferences follow a transitive Bradley-Terry model. This assumption fails to capture the non-transitive nature of population…
Active Learning for Direct Preference Optimization
Authors: Branislav Kveton, Xintong Li, Julian McAuley et al. Published: 2025-03-03 arXiv: 2503.01076v1 PDF: Download
Abstract: Direct preference optimization (DPO) is a form of reinforcement learning from human feedback (RLHF) where the policy is learned directly from preferential feedback. Although many models of human preferences exist, the critical task of selecting the most informative feedback for training them is unde…
Distributionally Robust Reinforcement Learning with Human Feedback
Authors: Debmalya Mandal, Paulius Sasnauskas, Goran Radanovic Published: 2025-03-01 arXiv: 2503.00539v1 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has evolved to be one of the main methods for fine-tuning large language models (LLMs). However, existing RLHF methods are non-robust, and their performance deteriorates if the downstream task differs significantly from the preference dataset used in…
C2-DPO: Constrained Controlled Direct Preference Optimization
Authors: Kavosh Asadi, Julien Han, Idan Pipano et al. Published: 2025-02-22 arXiv: 2502.17507v2 PDF: Download
Abstract: Direct preference optimization (\texttt{DPO}) has emerged as a promising approach for solving the alignment problem in AI. In this paper, we make two counter-intuitive observations about \texttt{DPO}. First, we show that \texttt{DPO} loss could be derived by starting from an alternative optimization…
Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective
Authors: Ruichen Shao, Bei Li, Gangao Liu et al. Published: 2025-02-20 arXiv: 2502.14340v1 PDF: Download
Abstract: Direct Preference Optimization (DPO) has gained attention as an efficient alternative to reinforcement learning from human feedback (RLHF) for aligning large language models (LLMs) with human preferences. Despite its advantages, DPO suffers from a length bias, generating responses longer than those …
Sample-Efficient Reinforcement Learning from Human Feedback via Information-Directed Sampling
Authors: Han Qi, Haochen Yang, Qiaosheng Zhang et al. Published: 2025-02-08 arXiv: 2502.05434v3 PDF: Download
Abstract: We study the problem of reinforcement learning from human feedback (RLHF), a critical problem in training large language models, from a theoretical perspective. Our main contribution is the design of novel sample-efficient RLHF algorithms based on information-directed sampling (IDS), an online decis…
PILAF: Optimal Human Preference Sampling for Reward Modeling
Authors: Yunzhen Feng, Ariel Kwiatkowski, Kunhao Zheng et al. Published: 2025-02-06 arXiv: 2502.04270v1 PDF: Download
Abstract: As large language models increasingly drive real-world applications, aligning them with human values becomes paramount. Reinforcement Learning from Human Feedback (RLHF) has emerged as a key technique, translating preference data into reward models when oracle human values remain inaccessible. In pr…
Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms
Authors: Xuerui Su, Yue Wang, Jinhua Zhu et al. Published: 2025-02-05 arXiv: 2502.03095v1 PDF: Download
Abstract: With the rapid development of Large Language Models (LLMs), numerous Reinforcement Learning from Human Feedback (RLHF) algorithms have been introduced to improve model safety and alignment with human preferences. These algorithms can be divided into two main frameworks based on whether they require …
Best Policy Learning from Trajectory Preference Feedback
Authors: Akhil Agnihotri, Rahul Jain, Deepak Ramachandran et al. Published: 2025-01-31 arXiv: 2501.18873v3 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful approach for aligning generative models, but its reliance on learned reward models makes it vulnerable to mis-specification and reward hacking. Preference-based Reinforcement Learning (PbRL) offers a more robust alternative …
Curiosity-Driven Reinforcement Learning from Human Feedback
Authors: Haoran Sun, Yekun Chai, Shuohuan Wang et al. Published: 2025-01-20 arXiv: 2501.11463v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but often at the cost of reduced output diversity. This trade-off between diversity and alignment quality remains a significant challenge. Drawing inspiration from …
Influencing Humans to Conform to Preference Models for RLHF
Authors: Stephane Hatgis-Kessell, W. Bradley Knox, Serena Booth et al. Published: 2025-01-11 arXiv: 2501.06416v2 PDF: Download
Abstract: Designing a reinforcement learning from human feedback (RLHF) algorithm to approximate a human’s unobservable reward function requires assuming, implicitly or explicitly, a model of human preferences. A preference model that poorly describes how humans generate preferences risks learning a poor appr…
2024
Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
Authors: Yannick Metz, David Lindner, Raphaël Baur et al. Published: 2024-11-18 arXiv: 2411.11761v2 PDF: Download
Abstract: Reinforcement Learning from Human feedback (RLHF) has become a powerful tool to fine-tune or train agentic machine learning models. Similar to how humans interact in social contexts, we can use many types of feedback to communicate our preferences, intentions, and knowledge to an RL agent. However, …
Entropy Controllable Direct Preference Optimization
Authors: Motoki Omura, Yasuhiro Fujita, Toshiki Kataoka Published: 2024-11-12 arXiv: 2411.07595v2 PDF: Download
Abstract: In the post-training of large language models (LLMs), Reinforcement Learning from Human Feedback (RLHF) is an effective approach to achieve generation aligned with human preferences. Direct Preference Optimization (DPO) allows for policy training with a simple binary cross-entropy loss without a rew…
Aligning Visual Contrastive learning models via Preference Optimization
Authors: Amirabbas Afzali, Borna Khodabandeh, Ali Rasekh et al. Published: 2024-11-12 arXiv: 2411.08923v3 PDF: Download
Abstract: Contrastive learning models have demonstrated impressive abilities to capture semantic similarities by aligning representations in the embedding space. However, their performance can be limited by the quality of the training data and its inherent biases. While Preference Optimization (PO) methods su…
SEE-DPO: Self Entropy Enhanced Direct Preference Optimization
Authors: Shivanshu Shekhar, Shreyas Singh, Tong Zhang Published: 2024-11-06 arXiv: 2411.04712v2 PDF: Download
Abstract: Direct Preference Optimization (DPO) has been successfully used to align large language models (LLMs) according to human preferences, and more recently it has also been applied to improving the quality of text-to-image diffusion models. However, DPO-based methods such as SPO, Diffusion-DPO, and D3PO…
Unveiling the Role of Expert Guidance: A Comparative Analysis of User-centered Imitation Learning and Traditional Reinforcement Learning
Authors: Amr Gomaa, Bilal Mahdy Published: 2024-10-28 arXiv: 2410.21403v1 PDF: Download
Abstract: Integration of human feedback plays a key role in improving the learning capabilities of intelligent systems. This comparative study delves into the performance, robustness, and limitations of imitation learning compared to traditional reinforcement learning methods within these systems. Recognizing…
Uncertainty-Penalized Direct Preference Optimization
Authors: Sam Houliston, Alizée Pace, Alexander Immer et al. Published: 2024-10-26 arXiv: 2410.20187v1 PDF: Download
Abstract: Aligning Large Language Models (LLMs) to human preferences in content, style, and presentation is challenging, in part because preferences are varied, context-dependent, and sometimes inherently ambiguous. While successful, Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Opti…
MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models
Authors: Ziyu Liu, Yuhang Zang, Xiaoyi Dong et al. Published: 2024-10-23 arXiv: 2410.17637v1 PDF: Download
Abstract: Visual preference alignment involves training Large Vision-Language Models (LVLMs) to predict human preferences between visual inputs. This is typically achieved by using labeled datasets of chosen/rejected pairs and employing optimization algorithms like direct preference optimization (DPO). Existi…
A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
Authors: Wenyi Xiao, Zechuan Wang, Leilei Gan et al. Published: 2024-10-21 arXiv: 2410.15595v3 PDF: Download
Abstract: With the rapid advancement of large language models (LLMs), aligning policy models with human preferences has become increasingly critical. Direct Preference Optimization (DPO) has emerged as a promising approach for alignment, acting as an RL-free alternative to Reinforcement Learning from Human Fe…
Accelerated Preference Optimization for Large Language Model Alignment
Authors: Jiafan He, Huizhuo Yuan, Quanquan Gu Published: 2024-10-08 arXiv: 2410.06293v1 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal tool for aligning large language models (LLMs) with human preferences. Direct Preference Optimization (DPO), one of the most popular approaches, formulates RLHF as a policy optimization problem without explicitly estimating t…
TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weights
Authors: Aiwei Liu, Haoping Bai, Zhiyun Lu et al. Published: 2024-10-06 arXiv: 2410.04350v3 PDF: Download
Abstract: Direct Preference Optimization (DPO) has been widely adopted for preference alignment of Large Language Models (LLMs) due to its simplicity and effectiveness. However, DPO is derived as a bandit problem in which the whole response is treated as a single arm, ignoring the importance differences betwe…
Dual Active Learning for Reinforcement Learning from Human Feedback
Authors: Pangpang Liu, Chengchun Shi, Will Wei Sun Published: 2024-10-03 arXiv: 2410.02504v2 PDF: Download
Abstract: Aligning large language models (LLMs) with human preferences is critical to recent advances in generative artificial intelligence. Reinforcement learning from human feedback (RLHF) is widely applied to achieve this objective. A key step in RLHF is to learn the reward function from human feedback. Ho…
MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
Authors: Yekun Chai, Haoran Sun, Huang Fang et al. Published: 2024-10-03 arXiv: 2410.02743v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has demonstrated effectiveness in aligning large language models (LLMs) with human preferences. However, token-level RLHF suffers from the credit assignment problem over long sequences, where delayed rewards make it challenging for the model to disce…
Conceptual Belief-Informed Reinforcement Learning
Authors: Xingrui Gu, Chuyi Jiang, Laixi Shi Published: 2024-10-02 arXiv: 2410.01739v4 PDF: Download
Abstract: Reinforcement learning (RL) has achieved significant success but is hindered by inefficiency and instability, relying on large amounts of trial-and-error data and failing to efficiently use past experiences to guide decisions. However, humans achieve remarkably efficient learning from experience, at…
VickreyFeedback: Cost-efficient Data Construction for Reinforcement Learning from Human Feedback
Authors: Guoxi Zhang, Jiuding Duan Published: 2024-09-27 arXiv: 2409.18417v2 PDF: Download
Abstract: This paper addresses the cost-efficiency aspect of Reinforcement Learning from Human Feedback (RLHF). RLHF leverages datasets of human preferences over outputs of large language models (LLM)s to instill human expectations into LLMs. Although preference annotation comes with a monetized cost, the eco…
Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization
Authors: Kaden Uhlig, Joern Wuebker, Raphael Reinauer et al. Published: 2024-09-26 arXiv: 2409.17673v3 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying task-alignment to neural machine translation (NMT) addresses an…
Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
Authors: Qining Zhang, Lei Ying Published: 2024-09-25 arXiv: 2409.17401v2 PDF: Download
Abstract: Reward inference (learning a reward model from human preferences) is a critical intermediate step in the Reinforcement Learning from Human Feedback (RLHF) pipeline for fine-tuning Large Language Models (LLMs). In practice, RLHF faces fundamental challenges such as distribution shift, reward model ov…
Evaluating Defences against Unsafe Feedback in RLHF
Authors: Domenic Rosati, Giles Edkins, Harsh Raj et al. Published: 2024-09-19 arXiv: 2409.12914v3 PDF: Download
Abstract: While there has been progress towards aligning Large Language Models (LLMs) with human values and ensuring safe behaviour at inference time, safety guards can easily be removed when fine tuned on unsafe and harmful datasets. While this setting has been treated extensively, another popular training p…
Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
Authors: Jiayi Zhou, Jiaming Ji, Juntao Dai et al. Published: 2024-08-30 arXiv: 2409.00162v1 PDF: Download
Abstract: Aligning the behavior of Large language models (LLMs) with human intentions and values remains a critical challenge. Reinforcement learning from human feedback (RLHF) aligns LLMs by training a reward model (RM) on human preferences and fine-tuning the LLMs to maximize RM feedback. Despite its effect…
UNA: Unifying Alignments of RLHF/PPO, DPO and KTO by a Generalized Implicit Reward Function
Authors: Zhichao Wang, Bin Bi, Can Huang et al. Published: 2024-08-27 arXiv: 2408.15339v3 PDF: Download
Abstract: An LLM is pretrained on trillions of tokens, but the pretrained LLM may still generate undesired responses. To solve this problem, alignment techniques such as RLHF, DPO and KTO are proposed. However, these alignment techniques have limitations. For example, RLHF requires training the reward model a…
Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
Authors: Sriyash Poddar, Yanming Wan, Hamish Ivison et al. Published: 2024-08-19 arXiv: 2408.10075v1 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) is a powerful paradigm for aligning foundation models to human values and preferences. However, current RLHF techniques cannot account for the naturally occurring differences in individual human preferences across a diverse population. When these dif…
Neural Dueling Bandits: Preference-Based Optimization with Human Feedback
Authors: Arun Verma, Zhongxiang Dai, Xiaoqiang Lin et al. Published: 2024-07-24 arXiv: 2407.17112v2 PDF: Download
Abstract: Contextual dueling bandit is used to model the bandit problems, where a learner’s goal is to find the best arm for a given context using observed noisy human preference feedback over the selected arms for the past contexts. However, existing algorithms assume the reward function is linear, which can…
Data-Centric Human Preference with Rationales for Direct Preference Alignment
Authors: Hoang Anh Just, Ming Jin, Anit Sahu et al. Published: 2024-07-19 arXiv: 2407.14477v4 PDF: Download
Abstract: Aligning language models with human preferences through reinforcement learning from human feedback is crucial for their safe and effective deployment. The human preference is typically represented through comparison where one response is chosen over another for a given prompt. However, standard pref…
Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning
Authors: Yuheng Zhang, Dian Yu, Baolin Peng et al. Published: 2024-06-30 arXiv: 2407.00617v4 PDF: Download
Abstract: Reinforcement Learning with Human Feedback (RLHF) has achieved great success in aligning large language models (LLMs) with human preferences. Prevalent RLHF approaches are reward-based, following the Bradley-Terry (BT) model assumption, which may not fully capture the complexity of human preferences…
Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
Authors: Xin Lai, Zhuotao Tian, Yukang Chen et al. Published: 2024-06-26 arXiv: 2406.18629v1 PDF: Download
Abstract: Mathematical reasoning presents a significant challenge for Large Language Models (LLMs) due to the extensive and precise chain of reasoning required for accuracy. Ensuring the correctness of each reasoning step is critical. To address this, we aim to enhance the robustness and factuality of LLMs by…
Robust Reinforcement Learning from Corrupted Human Feedback
Authors: Alexander Bukharin, Ilgee Hong, Haoming Jiang et al. Published: 2024-06-21 arXiv: 2406.15568v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) provides a principled framework for aligning AI systems with human preference data. For various reasons, e.g., personal bias, context ambiguity, lack of training, etc, human annotators may give incorrect or inconsistent preference labels. To tackle t…
WPO: Enhancing RLHF with Weighted Preference Optimization
Authors: Wenxuan Zhou, Ravi Agrawal, Shujian Zhang et al. Published: 2024-06-17 arXiv: 2406.11827v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) is a promising solution to align large language models (LLMs) more closely with human values. Off-policy preference optimization, where the preference data is obtained from other models, is widely adopted due to its cost efficiency and scalability. H…
DIPPER: Direct Preference Optimization to Accelerate Primitive-Enabled Hierarchical Reinforcement Learning
Authors: Utsav Singh, Souradip Chakraborty, Wesley A. Suttle et al. Published: 2024-06-16 arXiv: 2406.10892v3 PDF: Download
Abstract: Learning control policies to perform complex robotics tasks from human preference data presents significant challenges. On the one hand, the complexity of such tasks typically requires learning policies to perform a variety of subtasks, then combining them to achieve the overall goal. At the same ti…
Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
Authors: Qining Zhang, Honghao Wei, Lei Ying Published: 2024-06-11 arXiv: 2406.07455v2 PDF: Download
Abstract: In this paper, we study reinforcement learning from human feedback (RLHF) under an episodic Markov decision process with a general trajectory-wise reward model. We developed a model-free RLHF best policy identification algorithm, called $\mathsf{BSAD}$, without explicit reward model inference, which…
Optimizing Autonomous Driving for Safety: A Human-Centric Approach with LLM-Enhanced RLHF
Authors: Yuan Sun, Navid Salami Pargoo, Peter J. Jin et al. Published: 2024-06-06 arXiv: 2406.04481v1 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) is popular in large language models (LLMs), whereas traditional Reinforcement Learning (RL) often falls short. Current autonomous driving methods typically utilize either human feedback in machine learning, including RL, or LLMs. Most feedback guides…
Adaptive Preference Scaling for Reinforcement Learning with Human Feedback
Authors: Ilgee Hong, Zichong Li, Alexander Bukharin et al. Published: 2024-06-04 arXiv: 2406.02764v1 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values by learning rewards from human preference data. Due to various reasons, however, such data typically takes the form of rankings over pairs of trajectory segments, which fails to capture th…
Group Robust Preference Optimization in Reward-free RLHF
Authors: Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas et al. Published: 2024-05-30 arXiv: 2405.20304v1 PDF: Download
Abstract: Adapting large language models (LLMs) for specific tasks usually involves fine-tuning through reinforcement learning with human feedback (RLHF) on preference data. While these data often come from diverse labelers’ groups (e.g., different demographics, ethnicities, company teams, etc.), traditional …
On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
Authors: Jiancong Xiao, Ziniu Li, Xingyu Xie et al. Published: 2024-05-26 arXiv: 2405.16455v2 PDF: Download
Abstract: Accurately aligning large language models (LLMs) with human preferences is crucial for informing fair, economically sound, and statistically efficient decision-making processes. However, we argue that the predominant approach for aligning LLMs with human preferences through a reward model — reinfor…
Multi-turn Reinforcement Learning from Preference Human Feedback
Authors: Lior Shani, Aviv Rosenberg, Asaf Cassel et al. Published: 2024-05-23 arXiv: 2405.14655v2 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) has become the standard approach for aligning Large Language Models (LLMs) with human preferences, allowing LLMs to demonstrate remarkable abilities in various tasks. Existing methods work by emulating the preferences at the single decision (turn) le…
Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
Authors: Keertana Chidambaram, Karthik Vinay Seetharaman, Vasilis Syrgkanis Published: 2024-05-23 arXiv: 2405.15065v2 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) has become central to aligning large language models with human values, typically by first learning a reward model from preference data which is then used to update the model with reinforcement learning. Recent alternatives such as Direct Preference …
The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedback
Authors: Ruitao Chen, Liwei Wang Published: 2024-05-18 arXiv: 2405.11226v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has contributed to performance improvements in large language models. To tackle its reliance on substantial amounts of human-labeled data, a successful approach is multi-task representation learning, which involves learning a high-quality, low-dimens…
RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
Authors: Chanwoo Park, Mingyang Liu, Dingwen Kong et al. Published: 2024-04-30 arXiv: 2405.00254v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has been an effective technique for aligning AI systems with human values, with remarkable successes in fine-tuning large-language models recently. Most existing RLHF paradigms make the underlying assumption that human preferences are relatively homo…
Filtered Direct Preference Optimization
Authors: Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai et al. Published: 2024-04-22 arXiv: 2404.13846v4 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) plays a crucial role in aligning language models with human preferences. While the significance of dataset quality is generally recognized, explicit investigations into its impact within the RLHF framework, to our knowledge, have been limited. This p…
ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback
Authors: Zhenyu Hou, Yilin Niu, Zhengxiao Du et al. Published: 2024-04-01 arXiv: 2404.00934v2 PDF: Download
Abstract: ChatGLM is a free-to-use AI service powered by the ChatGLM family of large language models (LLMs). In this paper, we present the ChatGLM-RLHF pipeline — a reinforcement learning from human feedback (RLHF) system — designed to enhance ChatGLM’s alignment with human preferences. ChatGLM-RLHF encompa…
Disentangling Length from Quality in Direct Preference Optimization
Authors: Ryan Park, Rafael Rafailov, Stefano Ermon et al. Published: 2024-03-28 arXiv: 2403.19159v2 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) has been a crucial component in the recent success of Large Language Models. However, RLHF is know to exploit biases in human preferences, such as verbosity. A well-formatted and eloquent answer is often more highly rated by users, even when it is le…
Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
Authors: Wei Shen, Xiaoying Zhang, Yuanshun Yao et al. Published: 2024-03-12 arXiv: 2403.07708v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) is the mainstream paradigm used to align large language models (LLMs) with human preferences. Yet existing RLHF heavily relies on accurate and informative reward models, which are vulnerable and sensitive to noise from various sources, e.g. human lab…
Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences
Authors: Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban et al. Published: 2024-03-04 arXiv: 2403.01857v2 PDF: Download
Abstract: In this paper, we take a step towards a deeper understanding of learning from human preferences by systematically comparing the paradigm of reinforcement learning from human feedback (RLHF) with the recently proposed paradigm of direct preference optimization (DPO). We focus our attention on the cla…
When Your AIs Deceive You: Challenges of Partial Observability in Reinforcement Learning from Human Feedback
Authors: Leon Lang, Davis Foote, Stuart Russell et al. Published: 2024-02-27 arXiv: 2402.17747v5 PDF: Download
Abstract: Past analyses of reinforcement learning from human feedback (RLHF) assume that the human evaluators fully observe the environment. What happens when human feedback is based only on partial observations? We formally define two failure cases: deceptive inflation and overjustification. Modeling the hum…
Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
Authors: Arash Ahmadian, Chris Cremer, Matthias Gallé et al. Published: 2024-02-22 arXiv: 2402.14740v2 PDF: Download
Abstract: AI alignment in the shape of Reinforcement Learning from Human Feedback (RLHF) is increasingly treated as a crucial ingredient for high performance large language models. Proximal Policy Optimization (PPO) has been positioned by recent literature as the canonical method for the RL part of RLHF. Howe…
Generalizing Reward Modeling for Out-of-Distribution Preference Learning
Authors: Chen Jia Published: 2024-02-22 arXiv: 2402.14760v2 PDF: Download
Abstract: Preference learning (PL) with large language models (LLMs) aims to align the LLMs’ generations with human preferences. Previous work on reinforcement learning from human feedback (RLHF) has demonstrated promising results in in-distribution PL. However, due to the difficulty of obtaining human feedba…
COPR: Continual Human Preference Learning via Optimal Policy Regularization
Authors: Han Zhang, Lin Gui, Yu Lei et al. Published: 2024-02-22 arXiv: 2402.14228v3 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) is commonly utilized to improve the alignment of Large Language Models (LLMs) with human preferences. Given the evolving nature of human preferences, continual alignment becomes more crucial and practical in comparison to traditional static alignment…
Active Preference Optimization for Sample Efficient RLHF
Authors: Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano et al. Published: 2024-02-16 arXiv: 2402.10500v3 PDF: Download
Abstract: Large Language Models (LLMs) aligned using Reinforcement Learning from Human Feedback (RLHF) have shown remarkable generation abilities in numerous tasks. However, collecting high-quality human preferences creates costly bottlenecks in practical deployments, and hence, training data are often budget…
RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models
Authors: Saeed Khaki, JinJin Li, Lan Ma et al. Published: 2024-02-15 arXiv: 2402.10038v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has been extensively employed to align large language models with user intent. However, proximal policy optimization (PPO) based RLHF is occasionally unstable requiring significant hyperparameter finetuning, and computationally expensive to maximize …
Reinforcement Learning from Human Feedback with Active Queries
Authors: Kaixuan Ji, Jiafan He, Quanquan Gu Published: 2024-02-14 arXiv: 2402.09401v2 PDF: Download
Abstract: Aligning large language models (LLM) with human preference plays a key role in building modern generative models and can be achieved by reinforcement learning from human feedback (RLHF). Despite their superior performance, current RLHF approaches often require a large amount of human-labelled prefer…
MaxMin-RLHF: Alignment with Diverse Human Preferences
Authors: Souradip Chakraborty, Jiahao Qiu, Hui Yuan et al. Published: 2024-02-14 arXiv: 2402.08925v2 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) aligns language models to human preferences by employing a singular reward model derived from preference data. However, such an approach overlooks the rich diversity of human preferences inherent in data collected from multiple users. In this work, w…
Corruption Robust Offline Reinforcement Learning with Human Feedback
Authors: Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban et al. Published: 2024-02-09 arXiv: 2402.06734v1 PDF: Download
Abstract: We study data corruption robustness for reinforcement learning with human feedback (RLHF) in an offline setting. Given an offline dataset of pairs of trajectories along with feedback about human preferences, an $\varepsilon$-fraction of the pairs is corrupted (e.g., feedback flipped or trajectory fe…
LiPO: Listwise Preference Optimization through Learning-to-Rank
Authors: Tianqi Liu, Zhen Qin, Junru Wu et al. Published: 2024-02-02 arXiv: 2402.01878v3 PDF: Download
Abstract: Aligning language models (LMs) with curated human feedback is critical to control their behaviors in real-world applications. Several recent policy optimization methods, such as DPO and SLiC, serve as promising alternatives to the traditional Reinforcement Learning from Human Feedback (RLHF) approac…
Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble
Authors: Shun Zhang, Zhenfang Chen, Sunli Chen et al. Published: 2024-01-30 arXiv: 2401.16635v3 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) is a widely adopted approach for aligning large language models with human values. However, RLHF relies on a reward model that is trained with a limited amount of human preference data, which could lead to inaccurate predictions. As a result, RLHF ma…
A Minimaximalist Approach to Reinforcement Learning from Human Feedback
Authors: Gokul Swamy, Christoph Dann, Rahul Kidambi et al. Published: 2024-01-08 arXiv: 2401.04056v2 PDF: Download
Abstract: We present Self-Play Preference Optimization (SPO), an algorithm for reinforcement learning from human feedback. Our approach is minimalist in that it does not require training a reward model nor unstable adversarial training and is therefore rather simple to implement. Our approach is maximalist in…
2023
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
Authors: Yuanzhao Zhai, Han Zhang, Yu Lei et al. Published: 2023-12-30 arXiv: 2401.00243v1 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) emerges as a promising paradigm for aligning large language models (LLMs). However, a notable challenge in RLHF is overoptimization, where beyond a certain threshold, the pursuit of higher rewards leads to a decline in human preferences. In this pape…
Preference as Reward, Maximum Preference Optimization with Importance Sampling
Authors: Zaifan Jiang, Xing Huang, Chao Wei Published: 2023-12-27 arXiv: 2312.16430v5 PDF: Download
Abstract: Preference learning is a key technology for aligning language models with human values. Reinforcement Learning from Human Feedback (RLHF) is a model-based algorithm to optimize preference learning, which first fits a reward model for preference scores and then optimizes the generating policy with an…
REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback
Authors: Souradip Chakraborty, Anukriti Singh, Amisha Bhaskar et al. Published: 2023-12-22 arXiv: 2312.14436v3 PDF: Download
Abstract: The effectiveness of reinforcement learning (RL) agents in continuous control robotics tasks is mainly dependent on the design of the underlying reward function, which is highly prone to reward hacking. A misalignment between the reward function and underlying human preferences (values, social norms…
Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning
Authors: Sabrina McCallum, Max Taylor-Davies, Stefano V. Albrecht et al. Published: 2023-12-07 arXiv: 2312.04736v1 PDF: Download
Abstract: Despite numerous successes, the field of reinforcement learning (RL) remains far from matching the impressive generalisation power of human behaviour learning. One possible way to help bridge this gap be to provide RL agents with richer, more human-like feedback expressed in natural language. To inv…
Nash Learning from Human Feedback
Authors: Rémi Munos, Michal Valko, Daniele Calandriello et al. Published: 2023-12-01 arXiv: 2312.00886v4 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has emerged as the main paradigm for aligning large language models (LLMs) with human preferences. Typically, RLHF involves the initial step of learning a reward model from human feedback, often expressed as preferences between pairs of text generati…
The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization
Authors: Sian Gooding, Hassan Mansoor Published: 2023-11-02 arXiv: 2311.04919v1 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) can be used to capture complex and nuanced properties of text generation quality. As a result, the task of text summarization has been identified as a good candidate for this process. In this paper, we explore how preference agreement impacts the eff…
The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback
Authors: Nathan Lambert, Roberto Calandra Published: 2023-10-31 arXiv: 2311.00168v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) more capable in complex settings. RLHF proceeds as collecting human preference data, training a reward model on said data, and optimizing a base ML model with respect to said re…
COPR: Continual Learning Human Preference through Optimal Policy Regularization
Authors: Han Zhang, Lin Gui, Yuanzhao Zhai et al. Published: 2023-10-24 arXiv: 2310.15694v5 PDF: Download
Abstract: The technique of Reinforcement Learning from Human Feedback (RLHF) is a commonly employed method to improve pre-trained Language Models (LM), enhancing their ability to conform to human preferences. Nevertheless, the current RLHF-based LMs necessitate full retraining each time novel queries or feedb…
Active teacher selection for reinforcement learning from human feedback
Authors: Rachel Freedman, Justin Svegliato, Kyle Wray et al. Published: 2023-10-23 arXiv: 2310.15288v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) enables machine learning systems to learn objectives from human feedback. A core limitation of these systems is their assumption that all feedback comes from a single human teacher, despite querying a range of distinct teachers. We propose the Hidden…
Contrastive Preference Learning: Learning from Human Feedback without RL
Authors: Joey Hejna, Rafael Rafailov, Harshit Sikchi et al. Published: 2023-10-20 arXiv: 2310.13639v3 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular paradigm for aligning models with human intent. Typically RLHF algorithms operate in two phases: first, use human preferences to learn a reward function and second, align the model by optimizing the learned reward via reinfor…
Learning Optimal Advantage from Preferences and Mistaking it for Reward
Authors: W. Bradley Knox, Stephane Hatgis-Kessell, Sigurdur Orn Adalgeirsson et al. Published: 2023-10-03 arXiv: 2310.02456v1 PDF: Download
Abstract: We consider algorithms for learning reward functions from human preferences over pairs of trajectory segments, as used in reinforcement learning from human feedback (RLHF). Most recent work assumes that human preferences are generated based only upon the reward accrued within those segments, or thei…
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
Authors: Stephen Casper, Xander Davies, Claudia Shi et al. Published: 2023-07-27 arXiv: 2307.15217v2 PDF: Download
Abstract: Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizi…
DIP-RL: Demonstration-Inferred Preference Learning in Minecraft
Authors: Ellen Novoseller, Vinicius G. Goecks, David Watkins et al. Published: 2023-07-22 arXiv: 2307.12158v1 PDF: Download
Abstract: In machine learning for sequential decision-making, an algorithmic agent learns to interact with an environment while receiving feedback in the form of a reward signal. However, in many unstructured real-world settings, such a reward signal is unknown and humans cannot reliably craft a reward signal…
Preference Ranking Optimization for Human Alignment
Authors: Feifan Song, Bowen Yu, Minghao Li et al. Published: 2023-06-30 arXiv: 2306.17492v2 PDF: Download
Abstract: Large language models (LLMs) often contain misleading content, emphasizing the need to align them with human values to ensure secure AI systems. Reinforcement learning from human feedback (RLHF) has been employed to achieve this alignment. However, it encompasses two main drawbacks: (1) RLHF exhibit…
RRHF: Rank Responses to Align Language Models with Human Feedback without tears
Authors: Zheng Yuan, Hongyi Yuan, Chuanqi Tan et al. Published: 2023-04-11 arXiv: 2304.05302v3 PDF: Download
Abstract: Reinforcement Learning from Human Feedback (RLHF) facilitates the alignment of large language models with human preferences, significantly enhancing the quality of interactions between humans and models. InstructGPT implements RLHF through several stages, including Supervised Fine-Tuning (SFT), rewa…
2022
Improving Multimodal Interactive Agents with Reinforcement Learning from Human Feedback
Authors: Josh Abramson, Arun Ahuja, Federico Carnevale et al. Published: 2022-11-21 arXiv: 2211.11602v1 PDF: Download
Abstract: An important goal in artificial intelligence is to create agents that can both interact naturally with humans and learn from their feedback. Here we demonstrate how to use reinforcement learning from human feedback (RLHF) to improve upon simulated, embodied agents trained to a base level of competen…
Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning
Authors: David Lindner, Mennatallah El-Assady Published: 2022-06-27 arXiv: 2206.13316v1 PDF: Download
Abstract: Reinforcement learning (RL) commonly assumes access to well-specified reward functions, which many practical applications do not provide. Instead, recently, more work has explored learning what to do from interacting with humans. So far, most of these approaches model humans as being (nosily) ration…