Papperoni

2022

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Open PDF Google Scholar

citations

Cite Score

70

AI summary

This paper presents a method for training helpful and harmless AI assistants using preference modeling and reinforcement learning from human feedback (RLHF), demonstrating improved performance on NLP evaluations, compatibility with specialized skills, and robustness with iterated online training.

Main Contributions

Introduces preference modeling (PMing) and reinforcement learning from human feedback (RLHF) to finetune language models as helpful and harmless assistants.
Shows that alignment training improves performance on almost all NLP evaluations and is compatible with specialized skills like Python coding and summarization.
Explores an iterated online training mode where PMs and RL policies are updated weekly with fresh human feedback data, leading to efficient dataset and model improvement.
Investigates the robustness of RLHF training and identifies a roughly linear relation between RL reward and the square root of KL divergence between the policy and its initialization.
The proposed methods lead to models that are robust to a mix of helpful and harmless training data, especially with increasing model size.

Abstract

We apply preference modeling and reinforcement learning from human feedback (RLHF) to finetune language models to act as helpful and harmless assistants. We find this alignment training improves performance on almost all NLP evaluations, and is fully compatible with training for specialized skills such as python coding and summarization. We explore an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data, efficiently improving our datasets and models. Finally, we investigate the robustness of RLHF training, and identify a roughly linear relation between the RL reward and the square root of the KL divergence between the policy and its initialization. Alongside our main results, we perform peripheral analyses on calibration, competing objectives, and the use of OOD detection, compare our models with human writers, and provide samples from our models using prompts appearing in recent related work.

Citation Graph

Loading graph...

References [63]

Sort:

Filter:

[1]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[2]Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov - 2017

10 papers in library cite

Very simple methodology and very well explained. I also liked that they did a good job on motivating the method.

[3]Training Language Models to Follow Instructions With Human Feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, C. Wainwright, Pamela Mishkin, Chiyuan Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, Ryan Lowe - 2022

11 papers in library cite

No new research here. Only true contribution is scaling RLHF to GPT 3.

[4]DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter

Thomas Wolf - 2019

6 papers in library cite

Surprisingly simple and amazing results!

[5]Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba - 2021

9 papers in library cite

Very nice read! Nothin new in the methodology but very thoughtful and thorough analysis.

[6]Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt - 2021

6 papers in library cite

Ok, good benchmark but nothing surprising. I bet it saturated soon after (it did)

[7]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

P. Lewis, Ethan Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, Martha Lewis, W. T. Yih, Tim Rocktaschel, Sebastian Riedel, K. Douwe - 2020

5 papers in library cite

It coined the term "RAG" but really it was a very different concept.

[8]Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei - 2020

12 papers in library cite

Very nice! An amazing contribution. Problem is, the paper is just like 3 pages of actual interesting content, and 10 pages of detailed results. Boring to read but very good otherwise.

[9]Think You Have Solved Question Answering? Try arc, the Ai2 Reasoning Challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord - 2018

5 papers in library cite

Meh, this is just the dataset definition. I don't see anything special, just a new data source. No new methodology or anything.

[10]Hellaswag: Can a Machine Really Finish Your Sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi - 2019

6 papers in library cite

It's very simple, just a revamp on the SWAG dataset. However, the authors had a lot of foresight into understanding the dataset saturation loop - made me wonder if we stopped using adversarial examples because they just don't make sense anymore.

[11]Learning to Summarize from Human Feedback

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano - 2020

10 papers in library cite

Very thoughtful on explaining data collection and worrying about future consequences. Seems much closer to the RLHF we do now vs. what is in the human preferences paper.

[12]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[13]WebGPT: Browser-Assisted Question-Answering With Human Feedback

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeffrey Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman - 2021

7 papers in library cite

TBH the nicest thing about this is the idea of using a text-based web browser. Other than that, just another application of RLHF + PPO.

[14]The LAMBADA dataset: Word Prediction Requiring a Broad Discourse Context

D. Paperno, German Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, Raquel Fernandez - 2016

12 papers in library cite

Very nice paper - very interesting methodology to building it and very good when they bring a dataset that is meant to make machines to fail

[15]Pytorch: An Imperative Style, High-Performance Deep Learning Library

A. Paszke, S. Gross, Francisco Massa, Adam Lerer, J. Bradbury, G. Chanan, T. Killeen, Zongyu Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Devito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, Jinze Bai, S. Chintala - 2019

2 papers in library cite

[16]On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?

E. M. Bender, T. Gebru, Angelina McMillan-Major, S. Shmitchell - 2021

5 papers in library cite

[17]Palm: Scaling Language Modeling With Pathways

Aakanksha Chowdhery, S. Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, A. Roberts, P. Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann - 2023

6 papers in library cite

Missing author list

[18]On the Opportunities and Risks of Foundation Models

2021

3 papers in library cite

[19]Concrete Problems in AI Safety

Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Christiano, John Schulman, D. Mane - 2016

6 papers in library cite

[20]TruthfulQA: Measuring How Models Mimic Human Falsehoods

Stephen Lin, Jacob Hilton, Owain Evans - 2022

4 papers in library cite

[21]REALM: Retrieval-Augmented Language Model Pre-Training

K. Guu, K. Lee, Z. Tung, P. Panupong, M. W. Chang - 2020

5 papers in library cite

I think it's famous

[22]Ethical and Social Risks of Harm From Language Models

L. Weidinger, J. Mellor, M. Rauh, C. Griffin, Jonathan Uesato, P. S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh - 2021

2 papers in library cite

Missing author list

[23]Improving language models by retrieving from trillions of tokens

2021

1 paper in library cites

Missing author list

[24]Scaling Language Models: Methods, Analysis & Insights from Training Gopher

2021

4 papers in library cite

Gopher: Massive scaling + Google Deepmind

[25]Aligning AI With Shared Human Values

Dan Hendrycks, Collin Burns, Steven Basart, A. Critch, Jeffrey Li, Dawn Song, Jacob Steinhardt - 2020

3 papers in library cite

[26]Scaling Laws for Autoregressive Generative Modeling

Tom Henighan, Jared Kaplan, M. Katz, Mark Chen, Christopher Hesse, J. Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, C. Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, Sam McCandlish - 2020

5 papers in library cite

[27]A General Language Assistant as a Laboratory for Alignment

Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova Dassarma, Nelson Elhage, Zac Hatfield Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Jared Kaplan - 2021

5 papers in library cite

Start of the Anthropic RLHF journey. Not sure if worth the read but it's an option.

[28]Lamda: Language Models for Dialog Applications

R. Thoppilan, D. D. Freitas, J. Hall, Noam Shazeer, A. Kulshreshtha, H. Cheng, A. Jin, T. Bos, L. Baker, Yulun Du, Yiwei Li, Honglak Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, Deli Chen, Yiheng Xu, Ziru Chen, A. Roberts, Maarten Bosma, Y. Zhou, C. C. Chang, I. Krivokon, W. Rusch, M. Pickett, K. S. M. Hellstern, M. R. Morris, T. Doshi, R. D. Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. Olson, A. Molina, E. H. John, Jaehoon Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. A. Arcas, C. Cui, M. Croak, E. Chi, Quoc Le - 2022

5 papers in library cite

[29]Red Teaming Language Models With Language Models

Ethan Perez, S. Huang, Francis Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. Mcaleese, Geoffrey Irving - 2022

2 papers in library cite

[30]Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations

Philippe Tillet, H. T. Kung, D. Cox - 2019

1 paper in library cites

[31]Teaching Language Models to Support Answers With Verified Quotes

J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, Francis Song, M. Chadwick, M. Glaese, S. Young, L. C. Gillingham, Geoffrey Irving, N. Mcaleese - 2022

3 papers in library cite

[32]Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

T. Mihaylov, Peter Clark, Tushar Khot, Ashish Sabharwal - 2018

6 papers in library cite

[33]A Constructive Prediction of the Generalization Error Across Scales

J. S. Rosenfeld, A. Rosenfeld, Yonatan Belinkov, N. Shavit - 2019

5 papers in library cite

[34]Scaling Laws for Transfer

Danny Hernandez, Jared Kaplan, Tom Henighan, Sam McCandlish - 2021

5 papers in library cite

[35]A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Net Works

Dan Hendrycks, Kevin Gimpel - 2017

4 papers in library cite

[36]BBQ: A Hand-Built Bias Benchmark for Question Answering

Alicia Parrish, Anna Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, S. Bowman - 2022

4 papers in library cite

[37]Beyond Human-Level Accuracy: Computational Challenges in Deep Learning

J. Hestness, N. Ardalani, G. Diamos - 2019

4 papers in library cite

[38]On Calibration of Modern Neural Networks

C. Guo, G. Pleiss, Y. S. Sun, K. Q. Weinberger - 2017

4 papers in library cite

[39]Recipes for Safety in Open-Domain Chatbots

Jiacheng Xu, D. Ju, M. Li, Y. Lan Boureau, Jason Weston, E. Dinan - 2020

4 papers in library cite

[40]Deep Neural Networks Are Easily Fooled: High Confidence Predictions for Unrec Ognizable Images

Anh Nguyen, Jason Yosinski, J. Clune - 2015

3 papers in library cite

[41]Process for Adapting Language Models to Society (PALMS) With Values-Targeted Datasets

I. Solaiman, C. Dennison - 2021

3 papers in library cite

[42]Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles

B. Lakshminarayanan, A. Pritzel, C. Blundell - 2017

3 papers in library cite

[43]Deep anomaly Detection With Outlier Exposure

Dan Hendrycks, Mantas Mazeika, T. Dietterich - 2019

2 papers in library cite

[44]Ethical Challenges in Data-Driven Dialogue Systems

P. Henderson, K. Sinha, N. A. Gontier, N. R. Ke, G. Fried, Ryan Lowe, J. Pineau - 2018

2 papers in library cite

[45]Predictability and Surprise in Large Generative Models

Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova Dassarma, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Benjamin Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield Dodds, Scott Johnston, Shauna Kravec, Neel Nanda, Kamal Ndousse, Catherine Olsson, Dario Amodei, Dario Amodei, Tom B. Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Jack Clark - 2022

2 papers in library cite

[46]The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

A. Pan, K. Bhatia, Jacob Steinhardt - 2022

2 papers in library cite

[47]A Simple and Effective Baseline for Out-of-Distribution Detection Using Abstention

S. Thulasidasan, S. Thapa, S. Dhaubhadel, G. Chennupati, T. Bhattacharya, J. Bilmes - 2021

1 paper in library cites

[48]A Simple Fix to Mahalanobis Distance for Improving Near-Ood Detection

J. Ren, Stanislav Fort, Joseph Liu, A. G. Roy, S. Padhy, B. Lakshminarayanan - 2021

1 paper in library cites

[49]A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks

K. Lee, K. Lee, Honglak Lee, J. Shin - 2018

1 paper in library cites

[50]Adversarial Vulnerability of powerful Near Out-of-Distribution Detection

Stanislav Fort - 2022

1 paper in library cites

[51]Contrastive Training for Improved Out-of-Distribution Detection

J. Winkens, R. Bunel, A. G. Roy, R. Stanforth, V. Natarajan, J. R. Ledsam, P. Macwilliams, P. Kohli, A. Karthikesalingam, S. Kohl, C. Cemgil, T. Eslami, S. M. Ali Eslami, O. Ronneberger - 2020

1 paper in library cites

[52]Delphi: Towards Machine Ethics and Norms

L. Jiang, J. D. Hwang, C. Bhagavatula, R. L. Bras, M. Forbes, B. Borchardt, J. Liang, Oren Etzioni, Maarten Sap, Yejin Choi - 2021

1 paper in library cites

[53]Effect of Scale on Catastrophic Forgetting in Neural Networks

V. V. Ramasesh, Aitor Lewkowycz, Ethan Dyer - 2022

1 paper in library cites

[54]Enhancing the Reliability of Out-of-Distribution Image Detection in Neural Networks

S. Liang, Yiwei Li, R. Srikant - 2017

1 paper in library cites

[55]Exploring the Limits of Out-of-Distribution Detection

Stanislav Fort, J. Ren, B. Lakshminarayanan - 2021

1 paper in library cites

[56]Fine-Tuning GPT-2 From Human Preferences

D. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Dario Amodei, Alec Radford, Paul Christiano, Geoffrey Irving - 2019

1 paper in library cites

[57]Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra - 2022

1 paper in library cites

[58]Hybrid Models for Open Set Recognition

Haowei Zhang, A. Li, J. Guo, Y. Guo - 2020

1 paper in library cites

[59]Hybrid Models With Deep and Invertible Features

E. Nalisnick, A. Matsukawa, Yee Whye Teh, D. Gorur, B. Lakshminarayanan - 2019

1 paper in library cites

[60]Objective Robustness in Deep Reinforcement Learning

J. Koch, L. Langosco, J. Pfau, J. Le, L. Sharkey - 2021

1 paper in library cites

[61]On-the-Fly Controlled Text Generation With Experts and Anti-Experts

A. Liu, Maarten Sap, X. Lu, Swabha Swayamdipta, C. Bhagavatula, Noah A. Smith, Yejin Choi - 2021

1 paper in library cites

[62]Simple and Principled Uncertainty Estimation With Deterministic Deep Learning via Distance Awareness

J. Z. Liu, Zongyu Lin, S. Padhy, D. Tran, T. B. Weiss, B. Lakshminarayanan - 2020

1 paper in library cites

[63]Unsolved Problems in ML Safety

Dan Hendrycks, Nicholas Carlini, John Schulman, Jacob Steinhardt - 2021

1 paper in library cites

Cited by

4

papers in your library

Cites

30

papers in your library

Read

on May 28, 2026

It's ok but it's too long. The idea of optimizing HH is nice but nothing new in terms of RLHF.

Tags

Vetto StudyRLHF

Paper Aliases

No aliases