Papperoni

2018

Supervising Strong Learners by Amplifying Weak Experts

Paul Christiano, Buck Shlegeris, Dario Amodei

Open PDF Google Scholar

citations

Cite Score

10

AI summary

This paper introduces Iterated Amplification, a training strategy that builds training signals for complex problems by combining solutions to simpler subproblems, showing efficient learning of complex behaviors in algorithmic environments without external reward functions.

Main Contributions

Proposed Iterated Amplification, a novel training strategy that progressively builds up a training signal for difficult problems by combining solutions to easier subproblems.
Showed that Iterated Amplification can efficiently learn complex behaviors in algorithmic environments.
Demonstrated that the method is related to Expert Iteration but does not rely on an external reward function.
Introduced a human-predictor H' to reduce the burden on the human expert and generate training data.
Applied the framework to five toy algorithmic tasks, achieving comparable learning speed to supervised learning from ground truth data with a modest slowdown.

Abstract

Many real world learning tasks involve complex or hard-to-specify objectives, and using an easier-to-specify proxy can lead to poor performance or misaligned behavior. One solution is to have humans provide a training signal by demonstrating or judging performance, but this approach fails if the task is too complicated for a human to directly evaluate. We propose Iterated Amplification, an alternative training strategy which progressively builds up a training signal for difficult problems by combining solutions to easier subproblems. Iterated Amplification is closely related to Expert Iteration (Anthony et al., 2017; Silver et al., 2017b), except that it uses no external reward function. We present results in algorithmic environments, showing that Iterated Amplification can efficiently learn complex behaviors.

Citation Graph

Loading graph...

References [23]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]Deep Reinforcement Learning From Human Preferences

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei - 2017

11 papers in library cite

Very nice idea overall!

[3]Pointer Networks

Oriol Vinyals, M. Fortunato, Navdeep Jaitly - 2015

10 papers in library cite

Cool concept. Nice that it works and can find good solutions for TSP.

[4]Neural GPUs Learn Algorithms

Lukasz Kaiser, Ilya Sutskever - 2016

5 papers in library cite

Results and architecture are nice but TBH it is very poorly written... Doesn't seem like a lot of impact either.

[5]Algorithms for Inverse Reinforcement Learning

Andrew Y. Ng, S. Russell - 2000

3 papers in library cite

[6]Concrete Problems in AI Safety

Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Christiano, John Schulman, D. Mane - 2016

6 papers in library cite

[7]AI Safety via Debate

Geoffrey Irving, Paul Christiano, Dario Amodei - 2018

8 papers in library cite

[8]Mastering the Game of Go Without Human Knowledge

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Yanru Chen, T. Lillicrap, F. Hui, L. Sifre, G. V. D. Driessche, T. Graepel, Demis Hassabis - 2017

7 papers in library cite

[9]Hybrid Computing Using a Neural Network With Dynamic External Memory

Alex Graves, G. Wayne, M. Reynolds, T. Harley, Ivo Danihelka, A. G. Barwinska, S. G. Colmenarejo, Edward Grefenstette, T. Ramalho, J. Agapiou, A. P. Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, Phil Blunsom, Koray Kavukcuoglu, Demis Hassabis - 2016

5 papers in library cite

[10]Mastering Chess and Shogi by Self-Play With a General Reinforcement Learning Algorithm

D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, Demis Hassabis - 2017

5 papers in library cite

[11]Superintelligence: Paths, Dangers, Strategies

N. Bostrom - 2014

5 papers in library cite

[12]Thinking Fast and Slow With Deep Learning and Tree Search

T. Anthony, Z. Tian, D. Barber - 2017

4 papers in library cite

[13]Cooperative Inverse Reinforcement Learning

D. H. Menell, S. Russell, P. Abbeel, A. D. Dragan - 2016

3 papers in library cite

[14]Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

Chelsea Finn, Sergey Levine, P. Abbeel - 2016

3 papers in library cite

[15]Interactive Learning From Policy-Dependent Human Feedback

J. Macglashan, M. K. Ho, R. Loftin, B. Peng, D. Roberts, M. E. Taylor, M. L. Littman - 2017

3 papers in library cite

[16]Interactively Shaping Agents via Human Reinforcement: The TAMER Framework

W. B. Knox, P. Stone - 2009

3 papers in library cite

[17]Making Neural Programming Architectures Generalize via Recursion

J. Cai, R. Shin, Dawn Song - 2017

3 papers in library cite

[18]The Surprising Creativity of Digital Evolution: A Collection of Anecdotes From the Evolutionary Computation and Artificial Life Research Communities

J. Lehman, J. Clune, D. Misevic, C. Adami, J. Beaulieu, P. J. Bentley, S. Bernard, G. Belson, D. M. Bryson, N. Cheney, A. Cully, S. Doncieux, F. C. Dyer, K. O. Ellefsen, R. Feldt, S. Fischer, S. Forrest, A. Frenoy, C. Gagne, L. L. Goff, L. M. Grabowski, B. Hodjat, Frank Hutter, L. Keller, C. Knibbe, P. Krcah, R. E. Lenski, Hod Lipson, R. Maccurdy, C. Maestre, R. Miikkulainen, S. Mitri, D. E. Moriarty, J. B. Mouret, Anh Nguyen, C. Ofria, M. Parizeau, D. Parsons, R. T. Pennock, W. F. Punch, T. S. Ray, M. Schoenauer, E. Shulte, K. Sims, K. O. Stanley, F. Taddei, D. Tarapore, S. Thibault, W. Weimer, R. Watson, Jason Yosinski - 2018

3 papers in library cite

[19]Apprenticeship Learning via Inverse Reinforcement Learning

P. Abbeel, A. Ng - 2004

2 papers in library cite

[20]Faulty Reward Functions in the Wild

Jack Clark, Dario Amodei - 2016

2 papers in library cite

[21]Online Human Training of a Myoelectric Prosthesis Controller via Actor-Critic Reinforcement Learning

P. M. Pilarski, M. R. Dawson, T. Degris, F. Fahimi, J. P. Carey, Richard S. Sutton - 2011

2 papers in library cite

[22]Divide and Conquer With Neural Networks

A. Nowak, J. Bruna - 2016

1 paper in library cites

[23]Neural Programmer: Inducing Latent Programs With Gradient Descent

Arvind Neelakantan, Quoc V. Le, Ilya Sutskever - 2015

1 paper in library cites

Cited by

7

papers in your library

Cites

7

papers in your library

Read

on May 27, 2026

Nice idea, but doesn't have any concrete implementations or proof that it works. Sounds too aspirational.

Tags

Paper Aliases

No aliases