Papperoni

2024

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman

Open PDF Google Scholar

citations

Cite Score

63

AI summary

This paper introduces GPQA, a challenging dataset of 448 multiple-choice questions across biology, physics, and chemistry, demonstrating that experts achieve 65% accuracy and state-of-the-art AI systems (GPT-4) achieve 39% accuracy, making it suitable for scalable oversight research.

Main Contributions

Introduced GPQA, a new dataset of 448 graduate-level multiple-choice questions in biology, physics, and chemistry.
Ensured questions are high-quality and extremely difficult, with domain experts achieving 65% accuracy and non-experts only 34%.
Demonstrated that state-of-the-art AI systems (GPT-4) achieve 39% accuracy on GPQA.
Designed GPQA to be "Google-proof" for highly skilled non-experts, making it suitable for scalable oversight experiments.

Abstract

We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are high-quality and extremely difficult: experts who have or are pursuing PhDs in the corresponding domains reach 65% accuracy (74% when discounting clear mistakes the experts identified in retrospect), while highly skilled non-expert validators only reach 34% accuracy, despite spending on average over 30 minutes with unrestricted access to the web (i.e., the questions are “Google-proof"). The questions are also difficult for state-of-the-art AI systems, with our strongest GPT-4–based baseline achieving 39% accuracy. If we are to use future AI systems to help us answer very hard questions—for example, when developing new scientific knowledge—we need to develop scalable oversight methods that enable humans to supervise their outputs, which may be difficult even if the supervisors are themselves skilled and knowledgeable. The difficulty of GPQA both for skilled non-experts and frontier AI systems should enable realistic scalable oversight experiments, which we hope can help devise ways for human experts to reliably get truthful information from AI systems that surpass human capabilities.

Citation Graph

Loading graph...

References [37]

Sort:

Filter:

[1]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

27 papers in library cite

Expanded wor2vec. Very nice overall.

[2]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

19 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[3]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

38 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[4]Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt - 2021

10 papers in library cite

Ok, good benchmark but nothing surprising. I bet it saturated soon after (it did)

[5]Deep Reinforcement Learning From Human Preferences

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei - 2017

12 papers in library cite

Very nice idea overall!

[6]Natural Questions: A Benchmark for Question Answering Research

T. Kwiatkowski, J. Palomaki, O. Rhinehart, Michael Collins, A. P. Parikh, C. Alberti, D. Epstein, Illia Polosukhin, M. Kelcey, Jacob Devlin, K. Lee, K. N. Toutanova, Llion Jones, M. W. Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, Slav Petrov - 2019

10 papers in library cite

The dataset and methodology is very nice - it's amazing to see how Google does the summaries in search. However, the paper is too complex with the math stuff - unnecessary.

[7]TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension

M. Joshi, E. Choi, D. Weld, Luke Zettlemoyer - 2017

19 papers in library cite

I like the way they collect the data, and I think this is a nice dataset. However, it seems like they didn't even try to make a good baseline.

[8]Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aman Gupta, Adria Garriga Alonso - 2022

11 papers in library cite

Nice initiative but too much focus on models and performance rather than the bench itself. Also, saturated right after

[9]Scalable Agent Alignment via Reward Modeling: A Research Direction

Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg - 2018

6 papers in library cite

Low 4. Good push and good direction, but nothing groundbreaking - other research was already around about reward modeling. Good for them that they pushed for it.

[10]What Will it Take to Fix Benchmarking in Natural Language Understanding?

Samuel R. Bowman, George E. Dahl - 2021

3 papers in library cite

It is a nice push but really there are no clear outcomes. The proposal is very shallow.

[11]Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Wentao Li, P. J. Liu - 2019

18 papers in library cite

44 pages; T5 paper - "unifying framework where all text-based NLP problems are cast as text-to-text tasks"

[12]GPT-4 Technical Report

Openai - 2023

14 papers in library cite

GPT 4

[13]Large Language Models Are Zero-Shot Reasoners

T. Kojima, Shixiang Shane Gu, M. Reid, Y. Matsuo, Y. Iwasawa - 2022

8 papers in library cite

REASONING!

[14]Concrete Problems in AI Safety

Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Christiano, John Schulman, D. Mane - 2016

7 papers in library cite

[15]Hotpotqa: A Dataset for Diverse, Explainable Multi-Hop Question Answering

Zhilin Yang, P. Qi, S. Zhang, Yoshua Bengio, W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning - 2018

5 papers in library cite

[16]Towards Understanding Sycophancy in Language Models

Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, Ethan Perez - 2023

1 paper in library cites

[17]Discovering Language Model Behaviors With Model-Written Evaluations

Ethan Perez, Sam Ringer, K. Lukosiute, K. Nguyen, E. Chen, S. Heiner, C. Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, B. Israel, B. Seethor, Cameron Mckinnon, Christopher Olah, Da Yan, Dario Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran Johnson, G. Khundadze, Jackson Kernion, J. Landis, Jamie Kerr, Jared Mueller, J. Hyun, Joshua Landau, Kamal Ndousse, L. Goldberg, Liane Lovitt, M. Lucas, Michael Sellitto, Mingchuan Zhang, N. Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova Dassarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen Lawton, Tom B. Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, R. Grosse, Danny Hernandez, Deep Ganguli, E. Hubinger, Nicholas Schiefer, Jared Kaplan - 2022

1 paper in library cites

[18]Measuring Progress on Scalable Oversight for Large Language Models

S. Bowman, J. Hyun, Ethan Perez, E. Chen, C. Pettit, S. Heiner, K. Lukosiute, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron Mckinnon, Christopher Olah, Dario Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, J. D. Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemi Mercado, Nova Dassarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen Lawton, Tom B. Brown, T. J. Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield Dodds, Benjamin Mann, Jared Kaplan - 2022

3 papers in library cite

[19]AI Safety via Debate

Geoffrey Irving, Paul Christiano, Dario Amodei - 2018

9 papers in library cite

[20]Llama 2: Open Foundation and Fine-Tuned Chat Models

Hugo Touvron, L. Martin, K. Stone, P. Albert, Amjad Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, Mark Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, Jaehoon Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Zhicheng Yan, I. Zarov, Y. Z. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, T. Scialom - 2023

8 papers in library cite

[21]DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs

D. Dua, Yuzhi Wang, P. Dasigi, G. Stanovsky, Shivalika Singh, Matt Gardner - 2019

5 papers in library cite

[22]Red Teaming Language Models With Language Models

Ethan Perez, S. Huang, Francis Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. Mcaleese, Geoffrey Irving - 2022

3 papers in library cite

[23]Siren's Song in the AI ocean: A Survey on Hallucination in Large Language Models

Y. Z. Zhang, Yiwei Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Z. Zhang, Yanru Chen, Lisa Wang, Anh Tuan Luu, W. Bi, F. Shi, Sherry Shi - 2023

2 papers in library cite

[24]Commonsenseqa: A Question Answering Challenge Targeting Commonsense Knowledge

A. Talmor, J. Herzig, N. Lourie, Jonathan Berant - 2019

4 papers in library cite

[25]ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning

W. Yu, Zhejun Jiang, Y. Dong, J. Feng - 2020

2 papers in library cite

[26]A Comprehensive Evaluation of Tool-Assisted Generation Strategies

A. Jacovi, A. Caciularu, J. Herzig, R. Aharoni, B. Bohnet, Mor Geva - 2023

1 paper in library cites

[27]AI Safety Needs Social Scientists

Geoffrey Irving, Amanda Askell - 2019

1 paper in library cites

[28]AI Safety via Market Making

E. Hubinger - 2020

1 paper in library cites

[29]Controlled Crowdsourcing for High-Quality QA-SRL annotation

P. Roit, A. Klein, D. Stepanov, J. Mamou, Julian Michael, G. Stanovsky, Luke Zettlemoyer, Ido Dagan - 2020

1 paper in library cites

[30]Debate Helps Supervise Unreliable Experts

Julian Michael, S. Mahdi, David Rein, Jackson Petty, Julien Dirani, Vishakh Padmakumar, Samuel R. Bowman - 2023

1 paper in library cites

[31]Expertqa: Expert-Curated Questions and Attributed Answers

C. Malaviya, S. Lee, S. Chen, E. Sieber, M. Yatskar, Dan Roth - 2023

1 paper in library cites

[32]Measuring and Narrowing the Compositionality Gap in Language Models

Ofir Press, Mingchuan Zhang, S. Min, Ludwig Schmidt, Noah A. Smith, Martha Lewis - 2022

1 paper in library cites

Missing author listMissing year

[33]Model Index for Researchers

1 paper in library cites

[34]QuALITY: Question Answering With Long Input Texts, Yes!

Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Anna Chen, Vishakh Padmakumar, J. Ma, Jana Thompson, He He, S. Bowman - 2022

1 paper in library cites

[35]SQUALITY: Building a Long-Document Summarization Dataset the Hard Way

A. Wang, Richard Yuanzhe Pang, Anna Chen, Jason Phang, Samuel R. Bowman - 2022

1 paper in library cites

[36]The Case for Aligning Narrowly Superhuman Models

A. Cotra - 2021

1 paper in library cites

[37]What Do Models Learn From Question Answering Datasets?

P. Sen, A. Saffari - 2020

1 paper in library cites

Cited by

4

papers in your library

Cites

23

papers in your library

Read

on June 7, 2026

Even though it's small, it's very well made!

Tags

canonBenchmark

Paper Aliases

No aliases