Papperoni

2018

Scalable Agent Alignment via Reward Modeling: A Research Direction

Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg

Open PDF Google Scholar

citations

Cite Score

27

AI summary

This paper outlines a research direction for solving the agent alignment problem by learning a reward function from user interaction and optimizing it with reinforcement learning, discussing challenges and mitigation strategies to build trust in resulting agents.

Main Contributions

Proposes reward modeling as a solution to the agent alignment problem, separating learning 'what to achieve' from 'how to achieve it'.
Introduces recursive reward modeling to scale evaluation to complex domains by having agents assist users in feedback.
Discusses challenges in scaling reward modeling: amount of feedback, feedback distribution, reward hacking, unacceptable outcomes, and reward-result gap.
Outlines approaches to mitigate challenges including online feedback, off-policy feedback, leveraging existing data, hierarchical feedback, natural language, model-based RL, side-constraints, adversarial training, uncertainty estimates, and inductive bias.
Proposes methods for establishing trust in aligned agents through design choices, testing, interpretability, formal verification, and theoretical guarantees.

Abstract

One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with reinforcement learning. We discuss the key challenges we expect to face when scaling reward modeling to complex and general domains, concrete approaches to mitigate these challenges, and ways to establish trust in the resulting agents.

Citation Graph

Loading graph...

References [184]

Sort:

Filter:

[1]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[2]Deep Learning

I. Goodfellow, Yoshua Bengio, Y. A. Courville, A. Aaron - 2016

5 papers in library cite

It's an awesome review for people that are not familiar with deep learning, but it's very basic as well. Not too relevant given the context of other papers I read, but huge impact in terms of scientific communication.

[3]Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov - 2017

10 papers in library cite

Very simple methodology and very well explained. I also liked that they did a good job on motivating the method.

[4]Human-Level Control Through Deep Reinforcement Learning

V. Mnih - 2015

9 papers in library cite

Maybe the first instance of RL with NNs! I am impressed that it seems quite straightforward to implement (maybe tricky to get working).

[5]BLUE: A Method for Automatic Evaluation of Machine Translation

K. Papineni, S. Roukos, T. Ward, Wei Jing Zhu - 2002

19 papers in library cite

Very cool idea. Simple yet very impactful!

[6]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[7]Rectified Linear Units Improve Restricted Boltzmann Machines

V. Nair, Geoffrey E. Hinton - 2010

18 papers in library cite

I hate when people introduce a new idea but don't care to explain it! This is terrible compared to bengio's paper.

[8]Explaining and Harnessing Adversarial Examples

Ian J. Goodfellow, J. Shlens, Christian Szegedy - 2015

4 papers in library cite

It feels that it is an extension of the previous paper, which is nice. It is less original but also easier to read!

[9]Mastering the Game of Go With Deep Neural Networks and Tree Search

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. V. D. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Ilya Sutskever, T. Lillicrap, M. Leach, Koray Kavukcuoglu, T. Graepel, Demis Hassabis - 2016

5 papers in library cite

Surprisingly simple. Years ago when I first heard of AlphaGo it seemed such a different/far away concept - "wow, it's AI!". Now that I understand this, it sounds like something I could have worked on (if I had my current knowledge)

[10]Intriguing Properties of Neural Networks

Rob Fergus - 2014

7 papers in library cite

Very nice, and the first to notice how flaky NNs are. I think the end they went overboard with math, but the rest of the paper is very good.

[11]Representation Learning: A Review and New Perspectives

Yoshua Bengio, Aaron Courville, A. A. Vincent, P. Pascal - 2013

4 papers in library cite

Good in spirit but is too long and covers a lot of things that are not relevant anymore.

[12]Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

R. Williams - 1992

11 papers in library cite

It's alright for formalizing the concept, but it's a bit boring and doesn't add a lot from the middle on. Focuses too much in reviewing existing techniques and in stochastic units.

[13]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[14]Deep Reinforcement Learning From Human Preferences

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei - 2017

11 papers in library cite

Very nice idea overall!

[15]Handwritten Digit Recognition With a Back-Propagation Network

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel - 1990

10 papers in library cite

Just a minor derivation of the previous paper, adding more layers to the net. Pretty much the same exact text than the other one. Uninteresting if you read the other.

[16]Understanding Deep Learning Requires Rethinking Generalization

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals - 2016

2 papers in library cite

I like the push of rethinking generalization, and very intuitive to read. It is a bit incredible that it took so long for people to realize this - seems so obvious!

[17]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

A bit hard to understand, but very nice idea.

[18]Supervising Strong Learners by Amplifying Weak Experts

Paul Christiano, Buck Shlegeris, Dario Amodei - 2018

7 papers in library cite

Nice idea, but doesn't have any concrete implementations or proof that it works. Sounds too aspirational.

[19]Reinforcement Learning: An Introduction

Richard S. Sutton, A. Barto - 1998

5 papers in library cite

Book

[20]Improved Techniques for Training GANs

T. Salimans, I. Goodfellow, Wojciech Zaremba, V. Cheung, Alec Radford, X. Chen - 2016

1 paper in library cites

[21]Trust Region Policy Optimization

John Schulman, Sergey Levine, P. Abbeel, Michael I. Jordan, P. Moritz - 2015

4 papers in library cite

[22]Algorithms for Inverse Reinforcement Learning

Andrew Y. Ng, S. Russell - 2000

3 papers in library cite

[23]Weight Uncertainty in Neural Network

C. Blundell, J. Cornebise, Koray Kavukcuoglu, Daan Wierstra - 2015

3 papers in library cite

Introduced Bayesian NNs. I am not tooo excited about it but sounds relevant

[24]Concrete Problems in AI Safety

Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Christiano, John Schulman, D. Mane - 2016

6 papers in library cite

[25]Generative Adversarial Imitation Learning

J. Ho, Stefano Ermon - 2016

3 papers in library cite

[26]Empirical Evaluation of Rectified Activations in Convolutional Network

B. Xu, N. Wang, T. Chen, M. Li - 2015

2 papers in library cite

Implement "random RELU"

[27]AI Safety via Debate

Geoffrey Irving, Paul Christiano, Dario Amodei - 2018

8 papers in library cite

[28]Dataset Shift in Machine Learning

J. Q. Candela, M. Sugiyama, A. Schwaighofer, N. D. Lawrence - 2009

4 papers in library cite

Book, 250 pages

[29]Practical Variational Inference for Neural Networks

Alex Graves - 2011

8 papers in library cite

[30]Mastering the Game of Go Without Human Knowledge

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Yanru Chen, T. Lillicrap, F. Hui, L. Sifre, G. V. D. Driessche, T. Graepel, Demis Hassabis - 2017

7 papers in library cite

[31]Hybrid Computing Using a Neural Network With Dynamic External Memory

Alex Graves, G. Wayne, M. Reynolds, T. Harley, Ivo Danihelka, A. G. Barwinska, S. G. Colmenarejo, Edward Grefenstette, T. Ramalho, J. Agapiou, A. P. Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, Phil Blunsom, Koray Kavukcuoglu, Demis Hassabis - 2016

5 papers in library cite

[32]Reward Learning From Human Preferences and Demonstrations in Atari

B. Ibarz, Jan Leike, T. Pohlen, Geoffrey Irving, Shane Legg, Dario Amodei - 2018

5 papers in library cite

[33]Superintelligence: Paths, Dangers, Strategies

N. Bostrom - 2014

5 papers in library cite

[34]A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Net Works

Dan Hendrycks, Kevin Gimpel - 2017

4 papers in library cite

[35]On Calibration of Modern Neural Networks

C. Guo, G. Pleiss, Y. S. Sun, K. Q. Weinberger - 2017

4 papers in library cite

[36]Reinforcement Learning With a Corrupted Reward Channel

Tom Everitt, V. Krakovna, L. Orseau, Shane Legg - 2017

4 papers in library cite

[37]Thinking Fast and Slow With Deep Learning and Tree Search

T. Anthony, Z. Tian, D. Barber - 2017

4 papers in library cite

[38]Universal Intelligence: A Definition of Machine Intelligence

Shane Legg, M. Hutter - 2007

4 papers in library cite

[39]Asynchronous Methods for Deep Reinforcement Learning

V. Mnih, A. P. Badia, M. Mirza, Alex Graves, T. Lillicrap, T. Harley, D. Silver, Koray Kavukcuoglu - 2016

3 papers in library cite

[40]Bandit Based Monte-Carlo Planning

L. Kocsis, C. Szepesvari - 2006

3 papers in library cite

[41]Cooperative Inverse Reinforcement Learning

D. H. Menell, S. Russell, P. Abbeel, A. D. Dragan - 2016

3 papers in library cite

[42]Corrigibility

N. Soares, B. Fallenstein, S. Armstrong, E. Yudkowsky - 2015

3 papers in library cite

[43]Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Yarin Gal, Zoubin Ghahramani - 2015

3 papers in library cite

[44]Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

Chelsea Finn, Sergey Levine, P. Abbeel - 2016

3 papers in library cite

[45]Interactive Learning From Policy-Dependent Human Feedback

J. Macglashan, M. K. Ho, R. Loftin, B. Peng, D. Roberts, M. E. Taylor, M. L. Littman - 2017

3 papers in library cite

[46]Interactively Shaping Agents via Human Reinforcement: The TAMER Framework

W. B. Knox, P. Stone - 2009

3 papers in library cite

[47]Making Neural Programming Architectures Generalize via Recursion

J. Cai, R. Shin, Dawn Song - 2017

3 papers in library cite

[48]Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning

J. Kreutzer, J. Uyheng, S. Riezler - 2018

3 papers in library cite

[49]Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles

B. Lakshminarayanan, A. Pritzel, C. Blundell - 2017

3 papers in library cite

[50]The Surprising Creativity of Digital Evolution: A Collection of Anecdotes From the Evolutionary Computation and Artificial Life Research Communities

J. Lehman, J. Clune, D. Misevic, C. Adami, J. Beaulieu, P. J. Bentley, S. Bernard, G. Belson, D. M. Bryson, N. Cheney, A. Cully, S. Doncieux, F. C. Dyer, K. O. Ellefsen, R. Feldt, S. Fischer, S. Forrest, A. Frenoy, C. Gagne, L. L. Goff, L. M. Grabowski, B. Hodjat, Frank Hutter, L. Keller, C. Knibbe, P. Krcah, R. E. Lenski, Hod Lipson, R. Maccurdy, C. Maestre, R. Miikkulainen, S. Mitri, D. E. Moriarty, J. B. Mouret, Anh Nguyen, C. Ofria, M. Parizeau, D. Parsons, R. T. Pennock, W. F. Punch, T. S. Ray, M. Schoenauer, E. Shulte, K. Sims, K. O. Stanley, F. Taddei, D. Tarapore, S. Thibault, W. Weimer, R. Watson, Jason Yosinski - 2018

3 papers in library cite

[51]A Study on Overfitting in Deep Reinforcement Learning

Chiyuan Zhang, Oriol Vinyals, Rémi Munos, Samy Bengio - 2018

2 papers in library cite

[52]Active Reinforcement Learning: Observing Rewards at a Cost

David Krueger, Jan Leike, Owain Evans, J. Salvatier - 2016

2 papers in library cite

[53]AI Safety Gridworlds

Jan Leike, Miljan Martic, V. Krakovna, P. A. Ortega, Tom Everitt, A. Lefrancq, L. Orseau, Shane Legg - 2017

2 papers in library cite

[54]Apprenticeship Learning via Inverse Reinforcement Learning

P. Abbeel, A. Ng - 2004

2 papers in library cite

[55]April: Active Preference Learning-Based Reinforcement Learning

R. Akrour, M. Schoenauer, M. Sebag - 2012

2 papers in library cite

[56]Constrained Policy Optimization

Josh Achiam, D. Held, A. Tamar, P. Abbeel - 2017

2 papers in library cite

[57]Domain Randomization for Transferring Deep Neural Networks From Simulation to the Real World

J. Tobin, R. Fong, Alex Ray, J. Schneider, Wojciech Zaremba, P. Abbeel - 2017

2 papers in library cite

[58]Faulty Reward Functions in the Wild

Jack Clark, Dario Amodei - 2016

2 papers in library cite

[59]Learning From Human-Generated Reward

W. B. Knox - 2012

2 papers in library cite

[60]Learning With Latent Language

Jacob Andreas, Dan Klein, Sergey Levine - 2018

2 papers in library cite

[61]Neural Arithmetic Logic Units

A. Trask, F. Hill, S. Reed, J. Rae, C. Dyer, Phil Blunsom - 2018

2 papers in library cite

[62]Occam's Razor Is Insufficient to Infer the Preferences of Irrational Agents

S. Armstrong, S. Mindermann - 2018

2 papers in library cite

[63]Preference-Based Reinforcement Learning: A Formal Framework and a Policy Iteration Algorithm

J. Furnkranz, Eyke Hüllermeier, W. C. Cheng, S. H. Park - 2012

2 papers in library cite

[64]Probabilistic Backpropagation for Scalable Learning of Bayesian neural Networks

J. M. H. Lobato, R. A. Adams - 2015

2 papers in library cite

[65]Programming by Feedback

R. Akrour, M. Schoenauer, M. Sebag, J. C. Souplet - 2014

2 papers in library cite

C. Watkins, Peter Dayan - 1992

2 papers in library cite

[67]Score-Based Inverse Reinforcement Learning

L. E. Asri, Bilal Piot, M. Geist, R. Laroche, O. Pietquin - 2016

2 papers in library cite

[68]The Basic AI Drives

S. Omohundro - 2008

2 papers in library cite

[69]Third-Person Imitation Learning

B. C. Stadie, P. Abbeel, Ilya Sutskever - 2017

2 papers in library cite

[70]Variational dropout and the Local Reparameterization Trick

D. Kingma, T. Salimans, M. Welling - 2015

2 papers in library cite

[71]A Comprehensive Survey on Safe Reinforcement Learning

J. Garcia, F. Fernandez - 2015

1 paper in library cites

[72]A Dual Approach to Scalable Verification of Deep Networks

K. Dvijotham, R. Stanforth, S. Gowal, T. Mann, P. Kohli - 2018

1 paper in library cites

[73]A Formal Solution to the Grain of Truth Problem

Jan Leike, J. Taylor, B. Fallenstein - 2016

1 paper in library cites

[74]A Note on the Inception Score

S. Barratt, R. Sharma - 2018

1 paper in library cites

[75]A Note on the Pure Theory of Consumer's Behaviour

P. A. Samuelson - 1938

1 paper in library cites

[76]A Survey of Preference-Based Reinforcement Learning Methods

C. Wirth, R. Akrour, G. Neumann, J. Furnkranz - 2017

1 paper in library cites

[77]A Survey of Robot Learning From Demonstration

B. D. Argall, S. Chernova, M. Veloso, B. Browning - 2009

1 paper in library cites

[78]Accurate Uncertainties for Deep Learning Using Calibrated Regression

V. Kuleshov, N. Fenner, Stefano Ermon - 2018

1 paper in library cites

[79]Active Learning

B. Settles - 2012

1 paper in library cites

[80]Active Reinforcement Learning With Monte-Carlo Tree Search

S. Schulze, Owain Evans - 2018

1 paper in library cites

[81]Adversarial Attacks on Neural Network Policies

S. Huang, N. Papernot, I. Goodfellow, Y. Duan, P. Abbeel - 2017

1 paper in library cites

[82]Adversarial Risk and the Dangers of Evaluating Against Weak Attacks

Jonathan Uesato, B. O'donoghue, A. V. D. Oord, P. Kohli - 2018

1 paper in library cites

[83]Agent Foundations for Aligning Machine Intelligence With Human Interests: A Technical Research Agenda

N. Soares, B. Fallenstein - 2017

1 paper in library cites

[84]Agent-Agnostic Human-in-the-Loop Reinforcement Learning

D. Abel, J. Salvatier, Andreas Stuhlmuller, Owain Evans - 2017

1 paper in library cites

[85]AGI safety Literature Review

Tom Everitt, G. Lea, M. Hutter - 2018

1 paper in library cites

[86]Alignment for Advanced Machine Learning Systems

J. Taylor, E. Yudkowsky, P. Lavictoire, A. Critch - 2016

1 paper in library cites

[87]Asymptotic Non-Learnability of Universal Agents With Computable Horizon Functions

L. Orseau - 2013

1 paper in library cites

[88]Bad Universal Priors and Notions of Optimality

Jan Leike, M. Hutter - 2015

1 paper in library cites

[89]Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning

Richard S. Sutton, D. Precup, Shivalika Singh - 1999

1 paper in library cites

[90]Bridging the Gap: Converting Human Advice Into Imagined Examples

E. Yeh, M. Gervasio, D. Sanchez, M. Crossley, K. Myers - 2018

1 paper in library cites

[91]Building Safe Artificial Intelligence: Specification, Robustness, and Assurance

P. A. Ortega, Vishal Maini, D. S. Team - 2018

1 paper in library cites

[92]Coherent Extrapolated Volition

E. Yudkowsky - 2004

1 paper in library cites

[93]Computational Complexity: A Modern Approach

S. Arora, B. Barak - 2009

1 paper in library cites

[94]Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks With Many More Parameters Than Training Data

G. K. Dziugaite, D. M. Roy - 2017

1 paper in library cites

[95]Concrete Dropout

Yarin Gal, J. Hron, Alex Kendall - 2017

1 paper in library cites

[96]Continuous Control With Deep Reinforcement Learning

T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, Daan Wierstra - 2015

1 paper in library cites

[97]Deep Bayesian Active Learning With Image Data

Yarin Gal, R. Islam, Zoubin Ghahramani - 2017

1 paper in library cites

[98]Deep Learning in the Wild

T. Stadelmann, M. Amirian, I. Arabaci, M. Arnold, G. F. Duivesteijn, I. Elezi, M. Geiger, S. Lorwald, B. B. Meier, K. Rombach, L. Tuggener - 2018

1 paper in library cites

[99]Deep Learning: A Critical Appraisal

G. Marcus - 2018

1 paper in library cites

[100]Deep Q-Learning From Demonstrations

T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, Bilal Piot, D. Horgan, J. Quan, A. Sendonaris, G. D. Arnold, J. Agapiou, J. Z. Leibo, A. Gruslys - 2018

1 paper in library cites

[101]Deep Reinforcement Learning Doesn't Work Yet

A. Irpan - 2018

1 paper in library cites

[102]Deep Reinforcement Learning From Policy-Dependent Human Feedback

D. Arumugam, J. K. Lee, S. Saskin, M. L. Littman - 2018

1 paper in library cites

[103]Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces

G. Warnell, N. Waytowich, V. Lawhern, P. Stone - 2017

1 paper in library cites

[104]Delusion, Survival, and Intelligent Agents

M. Ring, L. Orseau - 2011

1 paper in library cites

[105]Directions and Desiderata for AI Alignment

Paul Christiano - 2017

1 paper in library cites

[106]Distributed Distributional Deterministic Policy Gradients

G. B. Maron, M. W. Hoffman, D. Budden, W. Dabney, D. Horgan, D. Tb, A. Muldal, N. Heess, T. Lillicrap - 2018

1 paper in library cites

[107]Distributed Prioritized Experience Replay

D. Horgan, J. Quan, D. Budden, G. B. Maron, M. Hessel, H. V. Hasselt, D. Silver - 2018

1 paper in library cites

[108]Does Your Model Know the Digit 6 Is Not a Cat? A Less Biased Evaluation of "Outlier" Detectors

A. Shafaei, M. Schmidt, J. J. Little - 2018

1 paper in library cites

[109]Efficient Training of Artificial Neural Networks for Autonomous Navigation

D. A. Pomerleau - 1991

1 paper in library cites

[110]Embedded Agency

A. Demski, S. Garrabrant - 2018

1 paper in library cites

[111]Ethical Issues in Advanced Artificial Intelligence

N. Bostrom - 2003

1 paper in library cites

[112]Factored Cognition

Andreas Stuhlmuller - 2018

1 paper in library cites

[113]FeUdal Networks for Hierarchical Reinforcement Learning

A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, Koray Kavukcuoglu - 2017

1 paper in library cites

[114]Feudal Reinforcement Learning

Peter Dayan, Geoffrey Hinton - 1993

1 paper in library cites

[115]GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

M. Heusel, H. Ramsauer, Thomas Unterthiner, B. Nessler, Sepp Hochreiter - 2017

1 paper in library cites

[116]Gödel machines: Self-Referential Universal Problem Solvers Making Provably Optimal Self-Improvements

Jürgen Schmidhuber - 2007

1 paper in library cites

[117]Graying the Black Box: Understanding DQNs

T. Zahavy, N. B. Zrihem, S. Mannor - 2016

1 paper in library cites

[118]Human-in-the-Loop Interpretability Prior

I. Lage, A. S. Ross, B. Kim, S. J. Gershman, F. D. Velez - 2018

1 paper in library cites

[119]Human-Level Performance in First-Person Multiplayer Games With Population-Based Deep Reinforcement Learning

M. Jaderberg, W. M. Czarnecki, I. Dunning, L. Marris, G. Lever, A. G. Castaneda, C. Beattie, N. C. Rabinowitz, A. S. Morcos, A. Ruderman, N. Sonnerat, T. Green, L. Deason, J. Z. Leibo, D. Silver, Demis Hassabis, Koray Kavukcuoglu, T. Graepel - 2018

1 paper in library cites

[120]IMPALA: Scalable Distributed Deep-Rl With Importance Weighted Actor-Learner Architectures

L. Espeholt, H. Soyer, Rémi Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, Shane Legg, Koray Kavukcuoglu - 2018

1 paper in library cites

[121]In Two Moves, AlphaGo and Lee Sedol Redefined the Future

C. Metz - 2016

1 paper in library cites

[122]Inverse Reward Design

D. H. Menell, S. Milli, P. Abbeel, S. Russell, A. D. Dragan - 2017

1 paper in library cites

[123]Leakproofing the Singularity: Artificial Intelligence Confinement Problem

R. Yampolskiy - 2012

1 paper in library cites

[124]Learning Agents for Uncertain Environments

S. Russell - 1998

1 paper in library cites

[125]Learning the Preferences of Ignorant, Inconsistent Agents

Owain Evans, Andreas Stuhlmuller, N. D. Goodman - 2016

1 paper in library cites

[126]Learning to Follow Language Instructions With Adversarial Reward Induction

D. Bahdanau, F. Hill, Jan Leike, E. Hughes, P. Kohli, Edward Grefenstette - 2018

1 paper in library cites

[127]Learning What to Value

D. Dewey - 2011

1 paper in library cites

[128]Logical Induction

S. Garrabrant, T. B. Tilsen, A. Critch, N. Soares, J. Taylor - 2016

1 paper in library cites

[129]Low Impact Artificial Intelligences

S. Armstrong, B. Levinstein - 2017

1 paper in library cites

[130]Measuring and Avoiding Side Effects Using Relative Reachability

V. Krakovna, L. Orseau, Miljan Martic, Shane Legg - 2018

1 paper in library cites

[131]Minimax-Regret Querying on Side Effects for Safe Optimality in Factored Markov Decision Processes

S. Zhang, E. H. Durfee, Shivalika Singh - 2018

1 paper in library cites

[132]Motivated Value Selection for Artificial Agents

S. Armstrong - 2015

1 paper in library cites

[133]Motivating the Rules of the Game for Adversarial Example Research

J. Gilmer, R. P. Adams, I. Goodfellow, D. Andersen, George E. Dahl - 2018

1 paper in library cites

[134]Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

A. Athalye, Nicholas Carlini, D. Wagner - 2018

1 paper in library cites

[135]On a Formal Model of Safe and Scalable Self-Driving Cars

S. S. Shwartz, S. Shammah, A. Shashua - 2017

1 paper in library cites

[136]Online Learning With Gated Linear Networks

J. Veness, T. Lattimore, A. Bhoopchand, A. G. Barwinska, C. Mattern, P. Toth - 2017

1 paper in library cites

[137]Optimization Daemons

Arbital - 2016

1 paper in library cites

[138]Playing Hard Exploration Games by Watching YouTube

Y. Aytar, T. Pfaff, D. Budden, T. L. Paine, Zhengtao Wang, N. D. Freitas - 2018

1 paper in library cites

[139]Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping

A. Ng, D. Harada, S. Russell - 1999

1 paper in library cites

[140]Policy Shaping: Integrating Human Feedback With Reinforcement Learning

S. Griffith, K. Subramanian, J. Scholz, C. L. Isbell, A. L. Thomaz - 2013

1 paper in library cites

[141]Predicting Human Deliberative Judgments With Machine Learning

Owain Evans, Andreas Stuhlmuller, C. Cundy, R. Carey, Z. Kenton, T. Mcgrath, A. Schreiber - 2018

1 paper in library cites

[142]Recurrent Experience Replay in Distributed Reinforcement Learning

Anonymous - 2019

1 paper in library cites

B. Mulvaney - 2012

1 paper in library cites

[144]Reducibility Among Combinatorial Problems

R. Karp - 1972

1 paper in library cites

[145]Relational Inductive Biases, Deep Learning, and Graph Networks

P. W. Battaglia, J. B. Hamrick, V. Bapst, A. S. Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, Adam Santoro, R. Faulkner, C. G. Gulcehre, Francis Song, A. Ballard, J. Gilmer, G. Dahl, Ashish Vaswani, K. Allen, C. Nash, V. Langston, C. Dyer, N. Heess, Daan Wierstra, P. Kohli, M. Botvinick, Oriol Vinyals, Yiwei Li, Razvan Pascanu - 2018

1 paper in library cites

[146]Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks

G. Katz, C. Barrett, D. L. Dill, K. Julian, M. J. Kochenderfer - 2017

1 paper in library cites

[147]Repeated Inverse Reinforcement Learning

K. Amin, N. Jiang, Shivalika Singh - 2017

1 paper in library cites

[148]Research Priorities for Robust and Beneficial Artificial Intelligence

S. Russell, D. Dewey, M. Tegmark - 2015

1 paper in library cites

[149]Reward Learning From Narrated Demonstrations

H. Y. F. Tung, A. W. Harley, L. K. Huang, K. Fragkiadaki - 2018

1 paper in library cites

[150]Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures

Anonymous - 2019

1 paper in library cites

[151]Robust Cooperation in the Prisoner's Dilemma: Program Equilibrium via Provability Logic

M. Barasz, Paul Christiano, B. Fallenstein, M. Herreshoff, P. Lavictoire, E. Yudkowsky - 2014

1 paper in library cites

I. Asimov - 1942

1 paper in library cites

[153]Safely Interruptible Agents

L. Orseau, S. Armstrong - 2016

1 paper in library cites

[154]Scaling Provable Adversarial Defenses

E. Wong, F. Schmidt, J. H. Metzen, J. Zico Kolter - 2018

1 paper in library cites

[155]Should Robots Be Obedient?

S. Milli, D. H. Menell, A. D. Dragan, S. Russell - 2017

1 paper in library cites

[156]SNES Super Mario World (USA) "Arbitrary Code Execution" in 02:25.19

Masterjun - 2014

1 paper in library cites

[157]Social Choice and the Value Alignment Problem

M. Prasad - 2018

1 paper in library cites

[158]Social Choice Ethics in Artificial Intelligence

S. D. Baum - 2017

1 paper in library cites

[159]Some Moral and Technical Consequences of Automation

N. Wiener - 1960

1 paper in library cites

[160]Sources of Intuitions and Data on AGI

S. Garrabrant - 2018

1 paper in library cites

[161]Space-Time Embedded Intelligence

L. Orseau, M. Ring - 2012

1 paper in library cites

[162]Systematic Generalization: What Is Required and Can It Be Learned?

Anonymous - 2019

1 paper in library cites

[163]TCAV: Relative Concept Importance Testing With Linear Concept Activation Vectors

B. Kim, J. Gilmer, F. Viegas, U. Erlingsson, M. Wattenberg - 2017

1 paper in library cites

[164]Teachable Robots: Understanding Human Teaching Behavior to Build More Effective Robot Learners

A. L. Thomaz, C. Breazeal - 2008

1 paper in library cites

[165]The AGI Containment Problem

J. Babcock, J. Kramar, R. Yampolskiy - 2016

1 paper in library cites

[166]The Alignment Problem for Bayesian History-Based Reinforcement Learners

Tom Everitt, M. Hutter - 2018

1 paper in library cites

[167]The Building Blocks of Interpretability

Christopher Olah, A. Satyanarayan, I. Johnson, S. Carter, L. Schubert, K. Ye, A. Mordvintsev - 2018

1 paper in library cites

[168]The Expected-Outcome Model of Two-Player Games

B. Abramson - 1987

1 paper in library cites

[169]The First Law of Robotics (A Call to Arms)

Oren Etzioni, D. Weld - 1994

1 paper in library cites

[170]The Off-Switch Game

D. H. Menell, A. D. Dragan, P. Abbeel, S. Russell - 2017

1 paper in library cites

[171]The Value Learning Problem

N. Soares - 2015

1 paper in library cites

[172]Towards a Rigorous Science of Interpretable Machine Learning

F. D. Velez, B. Kim - 2017

1 paper in library cites

[173]Towards Deep Learning Models Resistant to Adversarial Attacks

A. Madry, A. Makelov, Ludwig Schmidt, D. Tsipras, A. Vladu - 2017

1 paper in library cites

[174]Towards Safe Artificial General Intelligence

Tom Everitt - 2018

1 paper in library cites

[175]Training Verified Learners With Learned Verifiers

K. Dvijotham, S. Gowal, R. Stanforth, R. Arandjelovic, B. O'donoghue, Jonathan Uesato, P. Kohli - 2018

1 paper in library cites

[176]Translating neuralese

Jacob Andreas, A. D. Dragan, Dan Klein - 2017

1 paper in library cites

[177]Trial Without Error: Towards Safe Reinforcement Learning via Human Intervention

William Saunders, Girish Sastry, A. Stuhlmueller, Owain Evans - 2018

1 paper in library cites

[178]Universal Artificial Intelligence

M. Hutter - 2005

1 paper in library cites

[179]Unrestricted Adversarial Examples

Tom B. Brown, Nicholas Carlini, Chiyuan Zhang, Catherine Olsson, Paul Christiano, I. Goodfellow - 2018

1 paper in library cites

[180]Using Stories to Teach Human Values to Artificial Agents

M. O. Riedl, B. Harrison - 2016

1 paper in library cites

[181]Verifiable Reinforcement Learning via Policy Extraction

O. Bastani, Y. Pu, A. S. Lezama - 2018

1 paper in library cites

[182]Visualizing and Understanding Atari Agents

S. Greydanus, A. Koul, J. Dodge, A. Fern - 2018

1 paper in library cites

[183]Watching All the Movies Ever Made

P. Peter - 2014

1 paper in library cites

[184]What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

Alex Kendall, Yarin Gal - 2017

1 paper in library cites

Cited by

5

papers in your library

Cites

28

papers in your library

Read

on May 23, 2026

Low 4. Good push and good direction, but nothing groundbreaking - other research was already around about reward modeling. Good for them that they pushed for it.

Tags

RLHFVetto Study

Paper Aliases

No aliases