Papperoni

2023

Direct Preference Optimization: Your Language Model Is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

citations

Cite Score

AI summary

This paper introduces Direct Preference Optimization (DPO), a novel RL-free algorithm for training language models that aligns with human preferences by solving the standard RLHF problem with a simple classification loss, achieving performance comparable to or better than existing methods like PPO-based RLHF on sentiment control, summarization, and dialogue tasks.

Main Contributions

Introduces Direct Preference Optimization (DPO), a novel RL-free algorithm for training language models from preferences.
Proposes a new parameterization of the reward model in RLHF that enables extraction of the optimal policy in closed form.
Simplifies the RLHF problem to a single classification loss, eliminating the need for sampling during fine-tuning or extensive hyperparameter tuning.
Demonstrates DPO's stability, performance, and computational lightness compared to existing RLHF methods.
Achieves comparable or better performance than PPO-based RLHF in controlling sentiment, summarization, and single-turn dialogue.

Abstract

While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the completely unsupervised nature of their training. Existing methods for gaining such steerability collect human labels of the relative quality of model generations and fine-tune the unsupervised LM to align with these preferences, often with reinforcement learning from human feedback (RLHF). However, RLHF is a complex and often unstable procedure, first fitting a reward model that reflects the human preferences, and then fine-tuning the large unsupervised LM using reinforcement learning to maximize this estimated reward without drifting too far from the original model. In this paper we introduce a new parameterization of the reward model in RLHF that enables extraction of the corresponding optimal policy in closed form, allowing us to solve the standard RLHF problem with only a simple classification loss. The resulting algorithm, which we call Direct Preference Optimization (DPO), is stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning. Our experiments show that DPO can fine-tune LMs to align with human preferences as well as or better than existing methods. Notably, fine-tuning with DPO exceeds PPO-based RLHF in ability to control sentiment of generations, and matches or improves response quality in summarization and single-turn dialogue while being substantially simpler to implement and train.

Citation Graph

Loading graph...

References [49]

Sort:

Filter:

[1]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite