Papperoni

2022

Solving Math Word Problems With Process- And Outcome-Based Feedback

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins

Open PDF Google Scholar

citations

Cite Score

29

AI summary

This paper compares process-based and outcome-based supervision for language models on the GSM8K math word problem dataset, finding that while both achieve similar final-answer error rates, process-based feedback or emulated process-based feedback via reward models significantly reduces trace errors, achieving new state-of-the-art results.

Main Contributions

First comprehensive comparison between process- and outcome-based approaches for language models on a natural language task (GSM8K).
Pure outcome-based supervision yields similar final-answer error rates with less label supervision.
Achieved new state-of-the-art final-answer error rate of 12.7% and reasoning error rate of 3.4%.
Demonstrated that process-based supervision or a reward model emulating it is necessary for low trace error.
Found that outcome-supervised reward models approximate process-based labels, helping to explain their effectiveness in reducing trace error.

Abstract

Recent work has shown that asking language models to generate reasoning steps improves performance on many reasoning tasks. When moving beyond prompting, this raises the question of how we should supervise such models: outcome-based approaches which supervise the final result, or process-based approaches which supervise the reasoning process itself? Differences between these approaches might naturally be expected not just in final-answer errors but also in reasoning errors, which can be difficult to detect and are problematic in many real-world domains such as education. We run the first comprehensive comparison between process- and outcome-based approaches trained on a natural language task, GSM8K. We find that pure outcome-based supervision produces similar final-answer error rates with less label supervision. However, for correct reasoning steps we find it necessary to use process-based supervision or supervision from learned reward models that emulate process-based feedback. In total, we improve the previous best results from 16.8% rightarrow 12.7% final-answer error and 14.0% rightarrow 3.4% reasoning error among final-answer-correct solutions.

Citation Graph

Loading graph...

References [64]

Sort:

Filter:

[1]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[2]Decoupled Weight Decay Regularization

I. Loshchilov, Frank Hutter - 2017

7 papers in library cite

Very nice! Very simple to understand despite the complex topic. Very good results also

[3]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[4]Training Language Models to Follow Instructions With Human Feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, C. Wainwright, Pamela Mishkin, Chiyuan Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, Ryan Lowe - 2022

11 papers in library cite

No new research here. Only true contribution is scaling RLHF to GPT 3.

[5]Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba - 2021

9 papers in library cite

Very nice read! Nothin new in the methodology but very thoughtful and thorough analysis.

[6]Training Verifiers to Solve Math Word Problems

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman - 2021

7 papers in library cite

Fun read. Simple and nice use of verifiers

[7]Deep Reinforcement Learning From Human Preferences

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei - 2017

11 papers in library cite

Very nice idea overall!

[8]Measuring Mathematical Problem Solving With the MATH Dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt - 2021

8 papers in library cite

Good idea in creating the dataset but it's just a regular and uninteresting dataset description paper

[9]Learning to Summarize from Human Feedback

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano - 2020

10 papers in library cite

Very thoughtful on explaining data collection and worrying about future consequences. Seems much closer to the RLHF we do now vs. what is in the human preferences paper.

[10]Neural Turing Machines

Alex Graves, G. Wayne, Ivo Danihelka - 2014

18 papers in library cite

This paper is amazing. If someone told me that NNs could use and address memory by position I wouldn't believe it worked. Very nice, but it's a shame that it's just a toy example.

[11]WebGPT: Browser-Assisted Question-Answering With Human Feedback

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeffrey Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman - 2021

7 papers in library cite

TBH the nicest thing about this is the idea of using a text-based web browser. Other than that, just another application of RLHF + PPO.

[12]Fine-Tuning Language Models From Human Preferences

Geoffrey Irving - 2020

7 papers in library cite

It's so simple how they do it, plus I absolutely LOVED the "challenges" section and how honest they were about it. This is true research!

[13]Supervising Strong Learners by Amplifying Weak Experts

Paul Christiano, Buck Shlegeris, Dario Amodei - 2018

7 papers in library cite

Nice idea, but doesn't have any concrete implementations or proof that it works. Sounds too aspirational.

[14]Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Jason Wei, Xinpeng Wang, Dale Schuurmans, Maarten Bosma, Fanyue Xia, E. Chi, Quoc V. Le, Denny Zhou - 2022

10 papers in library cite

CoT

[15]Palm: Scaling Language Modeling With Pathways

Aakanksha Chowdhery, S. Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, A. Roberts, P. Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann - 2023

6 papers in library cite

[16]Large Language Models Are Zero-Shot Reasoners

T. Kojima, Shixiang Shane Gu, M. Reid, Y. Matsuo, Y. Iwasawa - 2022

6 papers in library cite

REASONING!

[17]Concrete Problems in AI Safety

Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Christiano, John Schulman, D. Mane - 2016

6 papers in library cite

[18]Hotpotqa: A Dataset for Diverse, Explainable Multi-Hop Question Answering

Zhilin Yang, P. Qi, S. Zhang, Yoshua Bengio, W. Cohen, Ruslan Salakhutdinov, Christopher D. Manning - 2018

4 papers in library cite

[19]Self-Consistency Improves Chain of Thought Reasoning in Language Models

Xinpeng Wang, Jason Wei, Dale Schuurmans, Quoc Le, E. Chi, Denny Zhou - 2022

5 papers in library cite

Missing author list

[20]Training Compute-Optimal Large Language Models

2022

4 papers in library cite

"DeepMind’s research showed that we weren't just compute-constrained, but data-constrained."

[21]Solving Quantitative Reasoning Problems With Language Models

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, C. Anil, I. Schlag, T. G. Solo - 2022

3 papers in library cite

Introduced Minerva

[22]Mathqa: Towards Interpretable Math Word Problem Solving With Operation-Based Formalisms

A. Amini, S. Gabriel, P. Lin, R. K. Kedziorski, Yejin Choi, Hananneh Hajishirzi - 2019

6 papers in library cite

Missing author list

[23]Recursively Summarizing Books with Human Feedback

2021

2 papers in library cite

OpenAI showing that HF is scallable

[24]AI Safety via Debate

Geoffrey Irving, Paul Christiano, Dario Amodei - 2018

8 papers in library cite

[25]Universal Transformers

Mostafa Dehghani, S. Gouws, Oriol Vinyals, Jakob Uszkoreit, Lukasz Kaiser - 2018

6 papers in library cite

[26]Show Your Work: Scratchpads for Intermediate Computation With Language Models

Maxwell Nye, A. J. Andreassen, Guy Gur Ari, Henryk Michalewski, Jacob Austin, D. Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, D. Luan, Charles Sutton, Augustus Odena - 2021

5 papers in library cite

First reasoning?

[27]A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers

S. Y. Miao, C. C. Liang, K. Y. Su - 2020

3 papers in library cite

[28]Teaching Language Models to Support Answers With Verified Quotes

J. Menick, M. Trebacz, V. Mikulik, J. Aslanides, Francis Song, M. Chadwick, M. Glaese, S. Young, L. C. Gillingham, Geoffrey Irving, N. Mcaleese - 2022

3 papers in library cite

[29]Mastering the Game of Go Without Human Knowledge

D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Yanru Chen, T. Lillicrap, F. Hui, L. Sifre, G. V. D. Driessche, T. Graepel, Demis Hassabis - 2017

7 papers in library cite

[30]Alignment of Language Agents

Z. Kenton, Tom Everitt, L. Weidinger, I. Gabriel, V. Mikulik, Geoffrey Irving - 2021

4 papers in library cite

[31]Reinforcement Learning With a Corrupted Reward Channel

Tom Everitt, V. Krakovna, L. Orseau, Shane Legg - 2017

4 papers in library cite

[32]Thinking Fast and Slow With Deep Learning and Tree Search

T. Anthony, Z. Tian, D. Barber - 2017

4 papers in library cite

[33]Generative Language Modeling for Automated Theorem Proving

S. Polu, Ilya Sutskever - 2020

3 papers in library cite

[34]Learning to Automatically Solve Algebra Word Problems

Nate Kushman, Y. Artzi, Luke Zettlemoyer, R. Barzilay - 2014

3 papers in library cite

[35]Making Neural Programming Architectures Generalize via Recursion

J. Cai, R. Shin, Dawn Song - 2017

3 papers in library cite

[36]On the Advance of Making Language Models Better Reasoners

Yiwei Li, Zongyu Lin, S. Zhang, Q. Fu, Berlin Chen, J. G. Lou, Weizhu Chen - 2022

3 papers in library cite

[37]Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems

W. Ling, D. Yogatama, C. Dyer, Phil Blunsom - 2017

3 papers in library cite

[38]Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning

Antonia Creswell, M. Shanahan, Irina Higgins - 2022

3 papers in library cite

[39]Star: Bootstrapping Reasoning With Reasoning

E. Zelikman, Yonghui Wu, J. Mu, N. Goodman - 2022

3 papers in library cite

[40]True Few-Shot Learning With Language Models

Ethan Perez, Douwe Kiela, Kyunghyun Cho - 2021

3 papers in library cite

[41]Without Specific Countermeasures, the Easiest Path to Transformative AI Likely Leads to AI Takeover

A. Cotra - 2022

3 papers in library cite

[42]Adaptive Computation Time for Recurrent Neural Networks

Alex Graves - 2016

2 papers in library cite

[43]Neural Programmer-Interpreters

S. Reed, N. D. Freitas - 2015

2 papers in library cite

[44]Specification Gaming: The Flip Side of AI Ingenuity

V. Krakovna, Jonathan Uesato, V. Mikulik, M. Rahtz, Tom Everitt, Ramana Kumar, Z. Kenton, Jan Leike, Shane Legg - 2020

2 papers in library cite

[45]Supervise Process, Not Outcomes

Andreas Stuhlmuller, J. Byun - 2022

2 papers in library cite

[46]A Graph Placement Methodology for Fast Chip Design

Azalia Mirhoseini, Anna Goldie, M. Yazgan, J. W. Jiang, E. Songhori, Shijie Wang, Y. J. Lee, E. Johnson, O. Pathak, A. Nazi - 2021

1 paper in library cites

[47]An Investigation of Model-Free Planning

A. Guez, M. Mirza, K. Gregor, R. Kabra, S. Racaniere, T. Weber, D. Raposo, Adam Santoro, L. Orseau, T. Eccles - 2019

1 paper in library cites

[48]Avoiding Tampering Incentives in Deep RL via Decoupled Approval

Jonathan Uesato, Ramana Kumar, V. Krakovna, Tom Everitt, R. Ngo, Shane Legg - 2020

1 paper in library cites

[49]Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models

M. Rauh, J. Mellor, Jonathan Uesato, P. S. Huang, J. Welbl, L. Weidinger, S. Dathathri, A. Glaese, Geoffrey Irving, I. Gabriel - 2022

1 paper in library cites

[50]Did Aristotle Use a Laptop? A Question Answering Benchmark With Implicit Reasoning Strategies

Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, Jonathan Berant - 2021

1 paper in library cites

[51]Explaining Answers With Entailment Trees

B. Dalvi, P. A. Jansen, Oyvind Tafjord, Z. Xie, H. Smith, L. Pipatanangkura, Peter Clark - 2021

1 paper in library cites

[52]Language Model Cascades

David Dohan, Weixin Xu, Aitor Lewkowycz, Jacob Austin, D. Bieber, R. G. Lopes, Yonghui Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl Dickstein - 2022

1 paper in library cites

[53]Mastering atari, Go, Chess and Shogi by Planning With a Learned Model

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, Demis Hassabis, T. Graepel - 2020

1 paper in library cites

[54]Maximum a Posteriori Policy Optimisation

A. Abdolmaleki, J. T. Springenberg, Y. Tassa, Rémi Munos, N. Heess, M. Riedmiller - 2018

1 paper in library cites

[55]Measuring Systematic Generalization in Neural Proof Generation With Transformers

N. Gontier, K. Sinha, Siva Reddy, C. Pal - 2020

1 paper in library cites

[56]Minimum Bayes-Risk Decoding for Statistical Machine Translation

S. Kumar, W. Byrne - 2004

1 paper in library cites

[57]Neural Program Lattices

Chun-Liang Li, D. Tarlow, A. L. Gaunt, M. Brockschmidt, Nate Kushman - 2016

1 paper in library cites

[58]On the Foundations of Noise-Free Selective Classification

R. E. Yaniv - 2010

1 paper in library cites

[59]Proofwriter: Generating Implications, Proofs, and Abductive Statements Over Natural Language

Oyvind Tafjord, B. D. Mishra, Peter Clark - 2020

1 paper in library cites

[60]REALab: An Embedded Perspective on Tampering

Ramana Kumar, Jonathan Uesato, R. Ngo, Tom Everitt, V. Krakovna, Shane Legg - 2020

1 paper in library cites

[61]Selective Classification for Deep Neural Networks

Y. Geifman, R. E. Yaniv - 2017

1 paper in library cites

[62]Unsupervised Commonsense Question Answering With Self-Talk

V. Shwartz, P. West, R. L. Bras, C. Bhagavatula, Yejin Choi - 2020

1 paper in library cites

[63]Unsupervised Question Decomposition for Question Answering

Ethan Perez, P. Lewis, W. T. Yih, Kyunghyun Cho, Douwe Kiela - 2020

1 paper in library cites

[64]Weakly-Supervised Semantic Parsing With Abstract Examples

O. Goldman, V. Latcinnik, U. Naveh, A. Globerson, Jonathan Berant - 2017

1 paper in library cites

Cited by

4

papers in your library

Cites

26

papers in your library

Read

on May 31, 2026

Very low 3. Almost 2. Boring, too big, and nothing surprising. Results were proved (partially) wrong later. I think they just formalized ORM vs. PRM

Tags

Paper Aliases

No aliases