Papperoni

2021

WebGPT: Browser-Assisted Question-Answering With Human Feedback

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeffrey Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman

Open PDF Google Scholar

citations

Cite Score

56

AI summary

This paper introduces WebGPT, a GPT-3-based model fine-tuned for long-form question-answering using a text-based web-browsing environment and human feedback. It collects references while browsing to support answers and achieves human-competitive performance on the ELI5 dataset.

Main Contributions

Introduction of a text-based web-browsing environment for language models to interact with, improving retrieval and synthesis.
Generation of answers with explicit references extracted by the model from web pages, crucial for factual accuracy evaluation.
Collection of new datasets including human demonstrations of web browsing for answers and human comparisons of model-generated answers.
Application of behavior cloning, reward modeling, reinforcement learning, and rejection sampling for training GPT-3 on long-form QA.
Achievement of human-competitive performance on the ELI5 dataset, with the best model's answers preferred over human demonstrators 56% of the time and over highest-voted Reddit answers 69% of the time.

Abstract

We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the web. By setting up the task so that it can be performed by humans, we are able to train models on the task using imitation learning, and then optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. We train and evaluate our models on ELI5, a dataset of questions asked by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.

Citation Graph

Loading graph...

References [30]

Sort:

Filter:

[1]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[2]Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov - 2017

10 papers in library cite

Very simple methodology and very well explained. I also liked that they did a good job on motivating the method.

[3]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

P. Lewis, Ethan Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kuttler, Martha Lewis, W. T. Yih, Tim Rocktaschel, Sebastian Riedel, K. Douwe - 2020

5 papers in library cite

It coined the term "RAG" but really it was a very different concept.

[4]Learning to Summarize from Human Feedback

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano - 2020

10 papers in library cite

Very thoughtful on explaining data collection and worrying about future consequences. Seems much closer to the RLHF we do now vs. what is in the human preferences paper.

[5]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

18 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[6]Scalable Agent Alignment via Reward Modeling: A Research Direction

Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg - 2018

5 papers in library cite

Low 4. Good push and good direction, but nothing groundbreaking - other research was already around about reward modeling. Good for them that they pushed for it.

[7]Supervising Strong Learners by Amplifying Weak Experts

Paul Christiano, Buck Shlegeris, Dario Amodei - 2018

7 papers in library cite

Nice idea, but doesn't have any concrete implementations or proof that it works. Sounds too aspirational.

[8]TruthfulQA: Measuring How Models Mimic Human Falsehoods

Stephen Lin, Jacob Hilton, Owain Evans - 2022

4 papers in library cite

[9]REALM: Retrieval-Augmented Language Model Pre-Training

K. Guu, K. Lee, Z. Tung, P. Panupong, M. W. Chang - 2020

5 papers in library cite

I think it's famous

[10]On Faithfulness and Factuality in Abstractive Summarization

J. Maynez, Shashi Narayan, B. Bohnet, R. Mcdonald - 2020

6 papers in library cite

[11]Retrieval Augmentation Reduces Hallucination in Conversation

K. Shuster, S. Poff, Mark Chen, Douwe Kiela, Jason Weston - 2021

3 papers in library cite

[12]Automation Bias: A Systematic Review of Frequency, Effect Mediators, and Mitigators

K. Goddard, A. Roudsari, J. C. Wyatt - 2012

1 paper in library cites

[13]ELI5: Long Form Question Answering

A. Fan, Yacine Jernite, Ethan Perez, D. Grangier, Jason Weston, Michael Auli - 2019

4 papers in library cite

[14]Truthful AI: Developing and Governing AI That Does Not Lie

Owain Evans, O. C. Barratt, L. Finnveden, A. Bales, A. Balwit, P. Wills, L. Righetti, William Saunders - 2021

1 paper in library cites

[15]AI Safety via Debate

Geoffrey Irving, Paul Christiano, Dario Amodei - 2018

8 papers in library cite

[16]Boosting Search Engines With Interactive Agents

L. Adolphs, B. Boerschinger, C. Buck, M. C. Huebscher, M. Ciaramita, L. Espeholt, T. Hofmann, Y. Kilcher - 2021

1 paper in library cites

[17]Superintelligence: Paths, Dangers, Strategies

N. Bostrom - 2014

5 papers in library cite

[18]Acceleration of Stochastic Approximation by Averaging

B. T. Polyak, A. B. Juditsky - 1992

4 papers in library cite

[19]Building Watson: An Overview of the DeepQA Project

D. Ferrucci, E. Brown, J. C. Carroll, J. Fan, D. Gondek, A. A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, C. Welty - 2013

3 papers in library cite

[20]Dense Passage Retrieval for Open-Domain Question Answering

V. Karpukhin, B. Ouguz, S. Min, L. Y. Wu, S. Edunov, Deli Chen, W. T. Yih - 2020

3 papers in library cite

[21]Crystal Society. Crystal Trilogy

M. Harms - 2016

1 paper in library cites

[22]Framing Theory

D. Chong, J. N. Druckman - 2007

1 paper in library cites

[23]Hurdles to Progress in Long-Form Question Answering

K. Krishna, A. Roy, M. Iyyer - 2021

1 paper in library cites

[24]Interactive Machine Comprehension With Information Seeking Agents

X. Yuan, J. Fu, M. A. Cote, Yi Tay, C. Pal, A. Trischler - 2019

1 paper in library cites

[25]Learning to Navigate the Web

I. Gur, U. Rueckert, A. Faust, D. H. Tur - 2018

1 paper in library cites

[26]Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets

P. Lewis, P. Stenetorp, Sebastian Riedel - 2020

1 paper in library cites

[27]Rethinking Search: Making Experts Out of Dilettantes

D. Metzler, Yi Tay, D. Bahri, M. Najork - 2021

1 paper in library cites

[28]Think You Have Solved Direct-Answer Question Answering? Try ARC-DA, the direct-answer AI2 Reasoning Challenge

S. Bhakthavatsalam, Daniel Khashabi, Tushar Khot, B. D. Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Peter Clark - 2021

1 paper in library cites

[29]UnitedQA: A Hybrid Approach for Open Domain Question Answering

H. Cheng, Y. Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao - 2021

1 paper in library cites

[30]World of Bits: An Open-Domain Platform for Web-Based Agents

T. Shi, A. Karpathy, L. Fan, J. Hernandez, Percy Liang - 2017

1 paper in library cites

Cited by

7

papers in your library

Cites

15

papers in your library

Read

on May 26, 2026

TBH the nicest thing about this is the idea of using a text-based web browser. Other than that, just another application of RLHF + PPO.

Tags

Vetto StudyRLHF

Paper Aliases

No aliases