Papperoni

2021

Measuring Mathematical Problem Solving With the MATH Dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt

Open PDF Google Scholar

citations

Cite Score

74

AI summary

This paper introduces MATH, a new dataset of 12,500 challenging competition mathematics problems with step-by-step solutions, and AMPS, a large auxiliary pretraining dataset, finding that even enormous Transformer models struggle with mathematical problem-solving, indicating a need for new algorithmic advancements.

Main Contributions

Introduction of MATH, a new dataset of 12,500 challenging competition mathematics problems with full step-by-step solutions, for measuring mathematical problem-solving ability in machine learning models.
Contribution of AMPS, a large auxiliary pretraining dataset derived from Khan Academy and Mathematica, to help models learn the fundamentals of mathematics.
Demonstration that state-of-the-art Transformer models achieve relatively low accuracy on MATH, suggesting that simply scaling models will not be sufficient for strong mathematical reasoning.
Observation that having models generate step-by-step solutions during inference can degrade accuracy, but providing partial ground truth step-by-step solutions or training on solutions can improve performance.
Analysis showing that AMPS pretraining enables smaller models to perform comparably to significantly larger fine-tuned models, highlighting its value as a pretraining dataset.

Abstract

Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.

Citation Graph

Loading graph...

References [42]

Sort:

Filter:

[1]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[4]Decoupled Weight Decay Regularization

I. Loshchilov, Frank Hutter - 2017

7 papers in library cite

Very nice! Very simple to understand despite the complex topic. Very good results also

[5]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[6]BART: Denoising Sequence-to-Sequence Pre-Training for Natural Language Generation, Translation, and Comprehension

Martha Lewis, Yibo Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer - 2019

6 papers in library cite

I expected SO much less from this paper, but it is simple, intuitive, AND achieves SotA!

[7]Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba - 2021

9 papers in library cite

Very nice read! Nothin new in the methodology but very thoughtful and thorough analysis.

[8]Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt - 2021

6 papers in library cite

Ok, good benchmark but nothing surprising. I bet it saturated soon after (it did)

[9]Scaling Laws for Neural Language Models

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei - 2020

12 papers in library cite

Very nice! An amazing contribution. Problem is, the paper is just like 3 pages of actual interesting content, and 10 pages of detailed results. Boring to read but very good otherwise.

[10]Huggingface's Transformers: State-of-the-Art Natural Language Processing

Thomas Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, Sam Shleifer, P. V. Platen, C. Ma, Yacine Jernite, J. Plu, Chenfeng Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, Alexander M. Rush - 2019

7 papers in library cite

It's a solid paper, but these framework papers are usually boring... Yet, it's nice to see that they started with reduced scope (and how large it got now).

[11]Program Synthesis With Large Language Models

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le - 2021

4 papers in library cite

Boring read. No news. They lost to OpenAI and HumanEval.

[12]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

[13]Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Wentao Li, P. J. Liu - 2019

17 papers in library cite

44 pages; T5 paper - "unifying framework where all text-based NLP problems are cast as text-to-text tasks"

[14]DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen - 2020

4 papers in library cite

"One of the final "classic" NLP architecture refinements that dominated leaderboards"

[15]Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

Suchin Gururangan, A. Marasovic, Swabha Swayamdipta, K. Lo, I. Beltagy, D. Downey, Noah A. Smith - 2020

2 papers in library cite

[16]Mathqa: Towards Interpretable Math Word Problem Solving With Operation-Based Formalisms

A. Amini, S. Gabriel, P. Lin, R. K. Kedziorski, Yejin Choi, Hananneh Hajishirzi - 2019

6 papers in library cite

[17]Scaling Laws for Autoregressive Generative Modeling

Tom Henighan, Jared Kaplan, M. Katz, Mark Chen, Christopher Hesse, J. Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, C. Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, Sam McCandlish - 2020

5 papers in library cite

[18]Unifiedqa: Crossing Format Boundaries With a Single Qa System

Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hananneh Hajishirzi - 2020

5 papers in library cite

Not sure if I want to read it. only ~400 citations

[19]A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers

S. Y. Miao, C. C. Liang, K. Y. Su - 2020

3 papers in library cite

[20]Scaling Laws for Transfer

Danny Hernandez, Jared Kaplan, Tom Henighan, Sam McCandlish - 2021

5 papers in library cite

[21]A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Net Works

Dan Hendrycks, Kevin Gimpel - 2017

4 papers in library cite

[22]Universal Intelligence: A Definition of Machine Intelligence

Shane Legg, M. Hutter - 2007

4 papers in library cite

[23]Generative Language Modeling for Automated Theorem Proving

S. Polu, Ilya Sutskever - 2020

3 papers in library cite

[24]Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems

W. Ling, D. Yogatama, C. Dyer, Phil Blunsom - 2017

3 papers in library cite

[25]Analysing Mathematical Reasoning Abilities of Neural Models

D. Saxton, Edward Grefenstette, F. Hill, P. Kohli - 2019

2 papers in library cite

[26]Deep Learning for Symbolic Mathematics

G. Lample, F. Charton - 2020

2 papers in library cite

[27]Gamepad: A Learning Environment for Theorem Proving

Dong Huang, Prafulla Dhariwal, Dawn Song, Ilya Sutskever - 2019

2 papers in library cite

[28]How Well Do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation

Dong Huang, Sherry Shi, Chin Yew Lin, J. Yin, W. Ma - 2016

2 papers in library cite

[29]LogiQA: A Challenge Dataset for Machine Reading Comprehension With Logical Reasoning

Joseph Liu, L. Cui, Haozhe Liu, Dong Huang, Yuzhi Wang, Y. Z. Zhang - 2020

2 papers in library cite

[30]Unsupervised Representation Learning With Deep Convolutional Generative Adversarial Networks

Alec Radford, Luke Metz, S. Chintala - 2015

2 papers in library cite

[31]A Formal Definition of Intelligence Based on an Intensional Variant of Algorithmic Complexity

Jose Hernandez Orallo - 1998

1 paper in library cites

[32]A Promising Path Towards Autoformalization and General Artificial Intelligence

Christian Szegedy - 2020

1 paper in library cites

[33]Beyond the Turing Test

Jose Hernandez Orallo - 2000

1 paper in library cites

[34]Expbert: Representation Engineering With Natural Language Explanations

S. Murty, P. W. Koh, Percy Liang - 2020

1 paper in library cites

[35]Holist: An Environment for Machine Learning of Higher-Order Theorem Proving (Extended Version)

K. Bansal, S. Loos, M. N. Rabe, Christian Szegedy, S. Wilcox - 2019

1 paper in library cites

[36]How to Solve It

G. Polya - 1945

1 paper in library cites

[37]Improving Graph Neural Network Representations of Logical Formulae With Subgraph Pooling

M. Crouse, I. Abdelaziz, C. Cornelio, V. Thost, L. Wu, K. D. Forbus, A. Fokoue - 2019

1 paper in library cites

[38]Lime: Learning Inductive Bias for Primitives of Mathematical Reasoning

Yonghui Wu, M. Rabe, Wentao Li, Jimmy Lei Ba, R. B. Grosse, Christian Szegedy - 2021

1 paper in library cites

[39]Mathematical Reasoning via Self-Supervised Skip-Tree Training

M. N. Rabe, D. L. Lee, K. Bansal, Christian Szegedy - 2020

1 paper in library cites

[40]MathZero, the Classification Problem, and Set-Theoretic Type Theory

D. A. Mcallester - 2020

1 paper in library cites

[41]Modelling High-Level Mathematical Reasoning in Mechanised Declarative Proofs

Wentao Li, Longhui Yu, Yonghui Wu, L. C. Paulson - 2020

1 paper in library cites

[42]The Unreasonable Effectiveness of Mathematics in the Natural Sciences

E. Wigner - 1960

1 paper in library cites

Cited by

8

papers in your library

Cites

17

papers in your library

Read

on May 30, 2026

Good idea in creating the dataset but it's just a regular and uninteresting dataset description paper

Tags

Paper Aliases

No aliases