Papperoni

2023

Math-Shepherd: Verify and Reinforce LLMS Step-by-Step Without Human Annotations

Peng Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yiwei Li, Deli Chen, Yonghui Wu, Zhifang Sui

Open PDF Google Scholar

citations

Cite Score

34

AI summary

This paper introduces MATH-SHEPHERD, a process-oriented math reward model that assigns scores to each step of math problem solutions using automatically constructed supervision data, achieving significant accuracy improvements on GSM8K and MATH datasets with various LLMs.

Main Contributions

Propose MATH-SHEPHERD, an automatic process annotation framework for math reasoning tasks, eliminating the need for human annotations.
Introduce a novel definition of an intermediate step's quality based on its potential to deduce the correct final answer.
Demonstrate MATH-SHEPHERD's effectiveness in two scenarios: reranking LLM outputs for verification and reinforcing LLMs with step-by-step Proximal Policy Optimization (PPO).
Achieve exceptional performance with open-source LLMs, for instance, improving Mistral-7B accuracy to 84.1% on GSM8K and 33.0% on MATH via step-by-step PPO.
Further enhance accuracy to 89.1% on GSM8K and 43.5% on MATH through MATH-SHEPHERD verification, setting new benchmarks for open-source models without additional tools.

Abstract

In this paper, we present an innovative process-oriented math process reward model called MATH-SHEPHERD, which assigns a reward score to each step of math problem solutions. The training of MATH-SHEPHERD is achieved using automatically constructed process-wise supervision data, breaking the bottleneck of heavy reliance on manual annotation in existing work. We explore the effectiveness of MATH-SHEPHERD in two scenarios: 1) Verification: MATH-SHEPHERD is utilized for reranking multiple outputs generated by Large Language Models (LLMs); 2) Reinforcement Learning: MATH-SHEPHERD is employed to reinforce LLMs with step-by-step Proximal Policy Optimization (PPO). With MATH-SHEPHERD, a series of open-source LLMs demonstrates exceptional performance. For instance, the step-by-step PPO with MATH-SHEPHERD significantly improves the accuracy of Mistral-7B (77.9%→84.1% on GSM8K and 28.6%→33.0% on MATH). The accuracy can be further enhanced to 89.1% and 43.5% on GSM8K and MATH with the verification of MATH-SHEPHERD, respectively. We believe that automatic process supervision holds significant potential for the future evolution of LLMs.

Citation Graph

Loading graph...

References [46]

Sort:

Filter:

[1]Mastering the Game of Go With Deep Neural Networks and Tree Search

D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. V. D. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Ilya Sutskever, T. Lillicrap, M. Leach, Koray Kavukcuoglu, T. Graepel, Demis Hassabis - 2016

5 papers in library cite

Surprisingly simple. Years ago when I first heard of AlphaGo it seemed such a different/far away concept - "wow, it's AI!". Now that I understand this, it sounds like something I could have worked on (if I had my current knowledge)

[2]Training Verifiers to Solve Math Word Problems

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman - 2021

7 papers in library cite

Fun read. Simple and nice use of verifiers

[3]Measuring Mathematical Problem Solving With the MATH Dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt - 2021

8 papers in library cite

Good idea in creating the dataset but it's just a regular and uninteresting dataset description paper

[4]Let's Verify Step by Step

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe - 2023

4 papers in library cite

Fun read but nothing really surprising in terms of methodology. Just proving PRMs are better than ORM (again, not surprising)

[5]Solving Math Word Problems With Process- And Outcome-Based Feedback

Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, Irina Higgins - 2022

4 papers in library cite

Very low 3. Almost 2. Boring, too big, and nothing surprising. Results were proved (partially) wrong later. I think they just formalized ORM vs. PRM

[6]Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Jason Wei, Xinpeng Wang, Dale Schuurmans, Maarten Bosma, Fanyue Xia, E. Chi, Quoc V. Le, Denny Zhou - 2022

10 papers in library cite

CoT

[7]GPT-4 Technical Report

Openai - 2023

6 papers in library cite

GPT 4

[8]Judging LLM-as-a-Judge With Mt-Bench and Chatbot arena

L. Zheng, Wei-Lin Chiang, Y. Sheng, S. Zhuang, Ziyi Wu, Y. Zhuang, Zongyu Lin, Zhiyuan Li, Dustin Li, E. Xing - 2023

1 paper in library cites

[9]Efficient Memory Management for Large Language Model Serving With Pagedattention

W. Kwon, Zhiyuan Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, Joseph Gonzalez, Haowei Zhang, Ion Stoica - 2023

5 papers in library cite

[10]Sparks of Artificial General Intelligence: Early Experiments With GPT-4

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Yiwei Li, S. Lundberg - 2023

3 papers in library cite

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. D. 1. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier - 2023

2 papers in library cite

[12]Self-Consistency Improves Chain of Thought Reasoning in Language Models

Xinpeng Wang, Jason Wei, Dale Schuurmans, Quoc Le, E. Chi, Denny Zhou - 2022

5 papers in library cite

[13]DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen - 2020

4 papers in library cite

"One of the final "classic" NLP architecture refinements that dominated leaderboards"

[14]Llama 2: Open Foundation and Fine-Tuned Chat Models

Hugo Touvron, L. Martin, K. Stone, P. Albert, Amjad Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, Mark Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, Jaehoon Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Zhicheng Yan, I. Zarov, Y. Z. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, T. Scialom - 2023

3 papers in library cite

[15]Bandit Based Monte-Carlo Planning

L. Kocsis, C. Szepesvari - 2006

3 papers in library cite

[16]Scaling Relationship on Learning Mathematical Reasoning With Large Language Models

Zhihao Yuan, H. Yuan, Chun-Liang Li, G. Dong, C. Tan, Chang Zhou - 2023

3 papers in library cite

[17]Tora: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

Z. Gou, Zhihong Shao, Y. Gong, Y. Shen, Yining Yang, M. Huang, N. Duan, Weizhu Chen - 2023

3 papers in library cite

[18]Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search

R. Coulom - 2006

2 papers in library cite

[19]Fast Inference From Transformers via Speculative Decoding

Y. Leviathan, M. Kalman, Y. Matias - 2023

2 papers in library cite

[20]Llemma: An Open Language Model for Mathematics

Z. Azerbayev, H. Schoelkopf, K. Paster, M. D. Santos, S. M. Mcaleer, A. Q. Jiang, J. Deng, Stella Biderman, S. Welleck - 2023

2 papers in library cite

[21]Making Large Language Models Better Reasoners With Alignment

Peng Wang, Lei Li, L. C. Chen, Francis Song, B. Lin, Yue Cao, T. Liu, Zhifang Sui - 2023

2 papers in library cite

[22]Mammoth: Building Math Generalist Models Through Hybrid Instruction Tuning

Xiang Yue, X. Qu, G. Zhang, Y. Fu, Weixiao Huang, Huan Sun, Yu Su, Weizhu Chen - 2023

2 papers in library cite

[23]Metamath: Bootstrap Your Own Mathematical Questions for Large Language Models

Longhui Yu, W. Jiang, H. Shi, J. Yu, Ze Liu, Y. Z. Zhang, J. T. Kwok, Zhiyuan Li, A. Weller, Weizhou Liu - 2023

2 papers in library cite

[24]Voyager: An Open-Ended Embodied Agent With Large Language Models

Gloria Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Yuxuan Zhu, L. Fan, A. Anandkumar - 2024

2 papers in library cite

[25]Wizardmath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct

H. Luo, Q. Sun, Chenfeng Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, Danyang Zhang - 2023

2 papers in library cite

[26]Challenges and Applications of Large Language Models

J. Kaddour, J. Harris, M. Mozes, H. Bradley, Roberta Raileanu, R. Mchardy - 2023

1 paper in library cites

[27]Complexity-Based Prompting for Multi-Step Reasoning

Y. Fu, H. Peng, Ashish Sabharwal, Peter Clark, Tushar Khot - 2022

1 paper in library cites

[28]Cumulative Reasoning With Large Language Models

Y. Z. Zhang, Jihan Yang, Y. Yuan, A. C. C. Yao - 2023

1 paper in library cites

[29]Deepseek llm: Let There Be Answers

Deepseek - 2023

1 paper in library cites

[30]Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

Ziyi Wu, Y. Hu, Weijia Shi, N. Dziri, A. Suhr, P. Ammanabrolu, Noah A. Smith, M. Ostendorf, Hananneh Hajishirzi - 2023

1 paper in library cites

[31]Generative Agents: Interactive Simulacra of Human Behavior

J. S. Park, J. O'brien, C. J. Cai, M. R. Morris, Percy Liang, M. S. Bernstein - 2023

1 paper in library cites

[32]Large Language Models Are Not Fair Evaluators

Peng Wang, Lei Li, L. C. Chen, D. Zhu, B. Lin, Yue Cao, Qian Liu, T. Liu, Zhifang Sui - 2023

1 paper in library cites

[33]Large Language Models Cannot Self-Correct Reasoning Yet

J. Huang, X. Chen, Swaroop Mishra, H. S. Zheng, A. W. Yu, X. Song, Denny Zhou - 2023

1 paper in library cites

[34]Let's Reinforce Step by Step

Siyuan Pan, V. Lialin, S. Muckatira, A. Rumshisky - 2023

1 paper in library cites

[35]Let's Reward Step by Step: Step-Level Reward Model as the Navigators for Reasoning

Q. Ma, H. Zhou, T. Liu, J. Yuan, P. Liu, Y. You, H. Yang - 2023

1 paper in library cites

[36]Lossless Speedup of Autoregressive Translation With Generalized Aggressive Decoding

H. Xia, Tiezheng Ge, F. Wei, Zhifang Sui - 2022

1 paper in library cites

[37]M3it: A Large-Scale Dataset Towards Multi-Modal Multilingual Instruction Tuning

Lei Li, Y. Yin, Shanda Li, L. C. Chen, Peng Wang, S. Ren, M. Li, Yining Yang, Jiacheng Xu, X. Sun - 2023

1 paper in library cites

[38]Making Language Models Better Reasoners With Step-Aware Verifier

Yiwei Li, Zongyu Lin, S. Zhang, Q. Fu, Berlin Chen, J. G. Lou, Weizhu Chen - 2023

1 paper in library cites

[39]Monte carlo Tree Search: A Review of Recent Modifications and Applications

M. Swiechowski, K. Godlewski, B. Sawicki, J. Mandziuk - 2023

1 paper in library cites

[40]Outcome-Supervised Verifiers for Planning in Mathematical Reasoning

F. Yu, A. Gao, B. Wang - 2023

1 paper in library cites

[41]Palm 2 Technical Report

R. Anil, A. M. Dai, O. Firat, M. J. Johnson, D. Lepikhin, A. Passos, Siamak Shakeri, E. Taropa, P. Bailey, Ziru Chen - 2023

1 paper in library cites

[42]Restgpt: Connecting Large Language Models With Real-World Applications via Restful Apis

Yueqi Song, W. Xiong, D. Zhu, Chun-Liang Li, K. Wang, Yuandong Tian, Shanda Li - 2023

1 paper in library cites

[43]Solving Math Word Problems via Cooperative Reasoning Induced Language Models

X. Zhu, J. Wang, Li Zhang, Y. Z. Zhang, Y. Huang, R. Gan, J. Zhang, Yining Yang - 2023

1 paper in library cites

[44]Towards End-to-End Embodied Decision Making via Multi-Modal Large Language Model: Explorations With Gpt4-Vision and Beyond

L. C. Chen, Y. Z. Zhang, S. Ren, H. Zhao, Zhipeng Cai, Yuzhi Wang, Peng Wang, T. Liu, B. Chang - 2023

1 paper in library cites

[45]Towards Reasoning in Large Language Models: A Survey

J. Huang, K. C. C. Chang - 2023

1 paper in library cites

[46]When Do Program-of-Thoughts Work for Reasoning?

Z. Bi, N. Zhang, Y. Jiang, S. Deng, G. Zheng, H. Chen - 2023

1 paper in library cites

Cited by

1

papers in your library

Cites

14

papers in your library

Read

on May 31, 2026

Low 4. Nice idea but boring paper. Would be nice if it had been used by more people though

Tags

Paper Aliases

No aliases