Papperoni

2017

Deep Reinforcement Learning From Human Preferences

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei

Open PDF Google Scholar

citations

Cite Score

79

AI summary

This paper introduces a deep reinforcement learning approach that uses non-expert human preferences between trajectory segments to solve complex RL tasks, including Atari games and simulated robot locomotion, with minimal human feedback.

Main Contributions

Demonstrates training deep RL systems using human preferences as a reward signal, significantly reducing the cost of human oversight.
Shows the ability to learn complex tasks like Atari games and simulated robot locomotion without direct access to a reward function.
Introduces an approach where human feedback is provided on less than 1% of the agent's interactions, making it practically applicable to state-of-the-art RL systems.
Successfully trains complex novel behaviors (e.g., robot backflips) with approximately one hour of human input.
Highlights the importance of online feedback collection to improve performance and prevent exploitation of learned reward function weaknesses.

Abstract

For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than 1% of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any which have been previously learned from human feedback.

Citation Graph

Loading graph...

References [45]

Sort:

Filter:

[1]Human-Level Control Through Deep Reinforcement Learning

V. Mnih - 2015

9 papers in library cite

Maybe the first instance of RL with NNs! I am impressed that it seems quite straightforward to implement (maybe tricky to get working).

[2]Mastering the Game of Go With Deep Neural Networks and Tree Search

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. V. D. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Ilya Sutskever, T. Lillicrap, M. Leach, Koray Kavukcuoglu, T. Graepel, Demis Hassabis - 2016

5 papers in library cite

Surprisingly simple. Years ago when I first heard of AlphaGo it seemed such a different/far away concept - "wow, it's AI!". Now that I understand this, it sounds like something I could have worked on (if I had my current knowledge)

[3]TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

M. Abadi, Akshat Agarwal, P. Barham, E. Brevdo, Ziru Chen, C. Citro, G. Corrado, A. Davis, Jeffrey Dean, M. Devin, Sanjay Ghemawat, I. Goodfellow, A. Harp, Geoffrey Irving, M. Isard, Y. Jia, R. Jozefowicz, Lukasz Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, Christopher Olah, M. Schuster, J. Shlens, B. Steiner, Ilya Sutskever, K. Talwar, P. Tucker, Vincent Vanhoucke, V. Vasudevan, F. Viegas, Oriol Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, Xiaoqiang Zheng - 2015

11 papers in library cite

This should be the golden standard to what framework papers should be. It's large, but it's not boring at all. Explains the core concepts while not going too deep as to describe unimportant things; explains design decisions and shortcomings... overall amazing

[4]Trust Region Policy Optimization

John Schulman, Sergey Levine, P. Abbeel, Michael I. Jordan, P. Moritz - 2015

4 papers in library cite

[5]Algorithms for Inverse Reinforcement Learning

Andrew Y. Ng, S. Russell - 2000

3 papers in library cite

[6]Concrete Problems in AI Safety

Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Christiano, John Schulman, D. Mane - 2016

6 papers in library cite

[7]Generative Adversarial Imitation Learning

J. Ho, Stefano Ermon - 2016

3 papers in library cite

[8]Superintelligence: Paths, Dangers, Strategies

N. Bostrom - 2014

5 papers in library cite

[9]The Arcade Learning Environment: An Evaluation Platform for General Agents

M. G. Bellemare, Y. Naddaf, J. Veness, M. Bowling - 2013

5 papers in library cite

[10]Asynchronous Methods for Deep Reinforcement Learning

V. Mnih, A. P. Badia, M. Mirza, Alex Graves, T. Lillicrap, T. Harley, D. Silver, Koray Kavukcuoglu - 2016

3 papers in library cite

[11]Cooperative Inverse Reinforcement Learning

D. H. Menell, S. Russell, P. Abbeel, A. D. Dragan - 2016

3 papers in library cite

[12]Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

Chelsea Finn, Sergey Levine, P. Abbeel - 2016

3 papers in library cite

[13]Interactive Learning From Policy-Dependent Human Feedback

J. Macglashan, M. K. Ho, R. Loftin, B. Peng, D. Roberts, M. E. Taylor, M. L. Littman - 2017

3 papers in library cite

[14]Interactively Shaping Agents via Human Reinforcement: The TAMER Framework

W. B. Knox, P. Stone - 2009

3 papers in library cite

[15]Learning Language Games Through Interaction

S. I. Wang, Percy Liang, Christopher D. Manning - 2016

3 papers in library cite

[16]Mujoco: A Physics Engine for Model-Based Control

E. Todorov, T. Erez, Y. Tassa - 2012

3 papers in library cite

Greg Brockman, V. Cheung, L. Pettersson, J. Schneider, John Schulman, Jie Tang, Wojciech Zaremba - 2016

3 papers in library cite

[18]Active Reinforcement Learning: Observing Rewards at a Cost

David Krueger, Jan Leike, Owain Evans, J. Salvatier - 2016

2 papers in library cite

[19]April: Active Preference Learning-Based Reinforcement Learning

R. Akrour, M. Schoenauer, M. Sebag - 2012

2 papers in library cite

[20]Individual Choice Behavior: A Theoretical Analysis

R. D. Luce - 2005

2 papers in library cite

[21]Learning From Human-Generated Reward

W. B. Knox - 2012

2 papers in library cite

[22]Online Human Training of a Myoelectric Prosthesis Controller via Actor-Critic Reinforcement Learning

P. M. Pilarski, M. R. Dawson, T. Degris, F. Fahimi, J. P. Carey, Richard S. Sutton - 2011

2 papers in library cite

[23]Playing Atari With Deep Reinforcement Learning

V. Mnih, Koray Kavukcuoglu, D. Silver, Alex Graves, I. Antonoglou, Daan Wierstra, M. Riedmiller - 2013

2 papers in library cite

[24]Preference-Based Reinforcement Learning: A Formal Framework and a Policy Iteration Algorithm

J. Furnkranz, Eyke Hüllermeier, W. C. Cheng, S. H. Park - 2012

2 papers in library cite

[25]Programming by Feedback

R. Akrour, M. Schoenauer, M. Sebag, J. C. Souplet - 2014

2 papers in library cite

[26]Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons

R. A. Bradley, M. E. Terry - 1952

2 papers in library cite

[27]Score-Based Inverse Reinforcement Learning

L. E. Asri, Bilal Piot, M. Geist, R. Laroche, O. Pietquin - 2016

2 papers in library cite

[28]Third-Person Imitation Learning

B. C. Stadie, P. Abbeel, Ilya Sutskever - 2017

2 papers in library cite

[29]A Bayesian Approach for Policy Learning From Trajectory Preference Queries

A. Wilson, A. Fern, P. Tadepalli - 2012

1 paper in library cites

[30]A Bayesian Interactive Optimization Approach to Procedural Animation Design

E. Brochu, T. Brochu, N. D. Freitas - 2010

1 paper in library cites

[31]Active Reward Learning

C. Daniel, M. Viering, J. Metz, O. Kroemer, J. Peters - 2014

1 paper in library cites

[32]Active Reward Learning With a Novel Acquisition Function

C. Daniel, O. Kroemer, M. Viering, J. Metz, J. Peters - 2015

1 paper in library cites

[33]Breeding a Diversity of super mario Behaviors Through Interactive Evolution

P. D. Sorensen, J. M. Olsen, S. Risi - 2016

1 paper in library cites

[34]Generalizing Skills With Semi-Supervised Reinforcement Learning

Chelsea Finn, Tao Yu, J. Fu, P. Abbeel, Sergey Levine - 2017

1 paper in library cites

[35]Introducing Machine Learning Within an Interactive Evolutionary Design Environment

A. Machwe, I. Parmee - 2006

1 paper in library cites

[36]Learning From Demonstrations for Real World Reinforcement Learning

T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, Bilal Piot, A. Sendonaris, G. D. Arnold, I. Osband, J. Agapiou, J. Z. Leibo, A. Gruslys - 2017

1 paper in library cites

[37]Learning Non-Myopically From Human-Generated Reward

W. B. Knox, P. Stone - 2013

1 paper in library cites

Author name contains 'et al'

[38]Model-Free Preference-Based Reinforcement Learning

C. Wirth, J. Furnkranz, G. Neumann, et al - 2016

1 paper in library cites

[39]Picbreeder: Evolving Pictures Collaboratively Online

J. Secretan, N. Beato, D. B. D. Ambrosio, A. Rodriguez, A. Campbell, K. O. Stanley - 2008

1 paper in library cites

[40]Preference-Based Policy Learning

R. Akrour, M. Schoenauer, M. Sebag - 2011

1 paper in library cites

[41]Preference-Based Reinforcement Learning: A Preliminary Survey

C. Wirth, J. Furnkranz - 2013

1 paper in library cites

[42]Preference-Learning Based Inverse Reinforcement Learning for Dialog Control

H. Sugiyama, T. Meguro, Y. Minami - 2012

1 paper in library cites

[43]Should We Fear Supersmart Robots?

S. Russell - 2016

1 paper in library cites

[44]Stimulus and Response Generalization: A Stochastic Model Relating Generalization to Distance in Psychological Space

R. N. Shepard - 1957

1 paper in library cites

[45]The Rating of Chessplayers, Past and Present

A. Elo - 1978

1 paper in library cites

Cited by

11

papers in your library

Cites

7

papers in your library

Read

on May 15, 2026

Very nice idea overall!

Tags

RLHF

Paper Aliases

No aliases