Papperoni

2024

Deepseekmath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Zhihong Shao, Peng Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Yiwei Li, Yonghui Wu

Open PDF Google Scholar

citations

Cite Score

76

AI summary

This paper introduces DeepSeekMath 7B, a language model pre-trained on 120B math-related tokens, achieving 51.7% on MATH benchmark without external tools and introduces Group Relative Policy Optimization (GRPO) for enhanced mathematical reasoning.

Main Contributions

Introduces DeepSeekMath 7B, an open language model that achieves 51.7% on the MATH benchmark without external tools or voting, comparable to Gemini-Ultra and GPT-4.
Develops the DeepSeekMath Corpus, a high-quality pre-training dataset of 120B math tokens extracted from Common Crawl using a fastText-based classifier and human annotation.
Presents Group Relative Policy Optimization (GRPO), an efficient variant of PPO that foregoes the critic model, reducing training resources and enhancing mathematical reasoning.
Demonstrates that code training prior to math training improves models' ability to solve mathematical problems.
Provides a unified paradigm for understanding various RL methods (RFT, DPO, PPO, GRPO) and conducts extensive experiments on their essential elements.

Abstract

Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.

Citation Graph

Loading graph...

References [59]

Sort:

Filter:

[1]Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov - 2017

10 papers in library cite

Very simple methodology and very well explained. I also liked that they did a good job on motivating the method.

[2]Decoupled Weight Decay Regularization

I. Loshchilov, Frank Hutter - 2017

7 papers in library cite

Very nice! Very simple to understand despite the complex topic. Very good results also

[3]Training Language Models to Follow Instructions With Human Feedback

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, C. Wainwright, Pamela Mishkin, Chiyuan Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, Ryan Lowe - 2022

11 papers in library cite

No new research here. Only true contribution is scaling RLHF to GPT 3.

[4]Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba - 2021

9 papers in library cite

Very nice read! Nothin new in the methodology but very thoughtful and thorough analysis.

[5]Direct Preference Optimization: Your Language Model Is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn - 2023

3 papers in library cite

Very nice idea and probably very revolutionary but paper is a bit boring.

[6]Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt - 2021

6 papers in library cite

Ok, good benchmark but nothing surprising. I bet it saturated soon after (it did)

[7]Training Verifiers to Solve Math Word Problems

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman - 2021

7 papers in library cite

Fun read. Simple and nice use of verifiers

[8]Measuring Mathematical Problem Solving With the MATH Dataset

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt - 2021

8 papers in library cite

Good idea in creating the dataset but it's just a regular and uninteresting dataset description paper

[9]Program Synthesis With Large Language Models

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le - 2021

4 papers in library cite

Boring read. No news. They lost to OpenAI and HumanEval.

[10]Let's Verify Step by Step

Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, Karl Cobbe - 2023

4 papers in library cite

Fun read but nothing really surprising in terms of methodology. Just proving PRMs are better than ORM (again, not surprising)

[11]Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, Jason Wei - 2022

4 papers in library cite

Good read and good analysis but no interesting research. They just distilled the benchmark and applied CoT. Good for showing that it works but not exciting.

[12]Math-Shepherd: Verify and Reinforce LLMS Step-by-Step Without Human Annotations

Peng Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yiwei Li, Deli Chen, Yonghui Wu, Zhifang Sui - 2023

1 paper in library cites

Low 4. Nice idea but boring paper. Would be nice if it had been used by more people though

[13]Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Jason Wei, Xinpeng Wang, Dale Schuurmans, Maarten Bosma, Fanyue Xia, E. Chi, Quoc V. Le, Denny Zhou - 2022

10 papers in library cite

CoT

[14]GPT-4 Technical Report

Openai - 2023

6 papers in library cite

GPT 4

[15]Gemini: A Family of Highly Capable Multimodal Models

R. Anil, S. Borgeaud, Yonghui Wu, J. Alayrac, J. Yu, Radu Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver, Slav Petrov, M. J. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, Jixuan Chen, E. Pitler, T. P. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan, B. Lee, F. Viola, M. Reynolds, Yiheng Xu, R. Doherty, E. Collins, C. M. Meyer, E. Rutherford, E. Moreira, K. Ayoub, M. Goel, G. Tucker, E. Piqueras, M. Krikun, I. Barr, N. Savinov, Ivo Danihelka, B. Roelofs, A. White, Anders Andreassen, T. V. Glehn, L. Yagati, M. Kazemi, L. Gonzalez, M. Khalman, J. Sygnowski - 2023

1 paper in library cites

[16]Efficient Memory Management for Large Language Model Serving With Pagedattention

W. Kwon, Zhiyuan Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, Joseph Gonzalez, Haowei Zhang, Ion Stoica - 2023

5 papers in library cite

A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. D. 1. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier - 2023

2 papers in library cite

[18]Fasttext. Zip: Compressing Text Classification Models

Armand Joulin, E. Grave, Piotr Bojanowski, M. Douze, Hervé Jégou, Tomas Mikolov - 2016

1 paper in library cites

[19]Solving Quantitative Reasoning Problems With Language Models

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, C. Anil, I. Schlag, T. G. Solo - 2022

3 papers in library cite

Introduced Minerva

[20]Llama 2: Open Foundation and Fine-Tuned Chat Models

Hugo Touvron, L. Martin, K. Stone, P. Albert, Amjad Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, Mark Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, Jaehoon Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Zhicheng Yan, I. Zarov, Y. Z. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, T. Scialom - 2023

3 papers in library cite

[21]Tree of Thoughts: Deliberate Problem Solving With Large Language Models

S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Yue Cao, K. Narasimhan - 2023

2 papers in library cite

[22]Deepseek LLM: Scaling Open-Source Language Models With Longtermism

Deepseek Ai - 2024

1 paper in library cites

[23]Deepseek-Coder: When the Large Language Model Meets Programming the Rise of Code Intelligence

Daniel Guo, Qihao Zhu, Diyi Yang, Z. Xie, K. Dong, Wenxuan Zhang, Guanduo Chen, Xiao Bi, Yonghui Wu, Y. K. Li, F. Luo, Yunyang Xiong, W. Liang - 2024

1 paper in library cites

[24]Program of Thoughts Prompting: Disentangling Computation From Reasoning for Numerical Reasoning Tasks

Weizhu Chen, X. Ma, Xinpeng Wang, W. W. Cohen - 2022

1 paper in library cites

[25]High-Dimensional Continuous Control Using Generalized Advantage Estimation

John Schulman, P. Moritz, Sergey Levine, M. Jordan, P. Abbeel - 2015

5 papers in library cite

[26]Generative Language Modeling for Automated Theorem Proving

S. Polu, Ilya Sutskever - 2020

3 papers in library cite

[27]Scaling Relationship on Learning Mathematical Reasoning With Large Language Models

Zhihao Yuan, H. Yuan, Chun-Liang Li, G. Dong, C. Tan, Chang Zhou - 2023

3 papers in library cite

[28]Tora: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

Z. Gou, Zhihong Shao, Y. Gong, Y. Shen, Yining Yang, M. Huang, N. Duan, Weizhu Chen - 2023

3 papers in library cite

[29]Approximating Kl Divergence

John Schulman - 2020

2 papers in library cite

[30]Fast Inference From Transformers via Speculative Decoding

Y. Leviathan, M. Kalman, Y. Matias - 2023

2 papers in library cite

[31]Language Models Are Multilingual Chain-of-Thought Reasoners

F. Shi, Mirac Suzgun, M. Freitag, Xinpeng Wang, S. Srivats, S. Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, Jason Wei - 2023

2 papers in library cite

[32]Llemma: An Open Language Model for Mathematics

Z. Azerbayev, H. Schoelkopf, K. Paster, M. D. Santos, S. M. Mcaleer, A. Q. Jiang, J. Deng, Stella Biderman, S. Welleck - 2023

2 papers in library cite

[33]Making Large Language Models Better Reasoners With Alignment

Peng Wang, Lei Li, L. C. Chen, Francis Song, B. Lin, Yue Cao, T. Liu, Zhifang Sui - 2023

2 papers in library cite

[34]Mammoth: Building Math Generalist Models Through Hybrid Instruction Tuning

Xiang Yue, X. Qu, G. Zhang, Y. Fu, Weixiao Huang, Huan Sun, Yu Su, Weizhu Chen - 2023

2 papers in library cite

[35]Metamath: Bootstrap Your Own Mathematical Questions for Large Language Models

Longhui Yu, W. Jiang, H. Shi, J. Yu, Ze Liu, Y. Z. Zhang, J. T. Kwok, Zhiyuan Li, A. Weller, Weizhou Liu - 2023

2 papers in library cite

[36]Solving Olympiad Geometry Without Human Demonstrations

T. H. Trinh, Yonghui Wu, Quoc V. Le, He He, T. Luong - 2024

2 papers in library cite

[37]Wizardmath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct

H. Luo, Q. Sun, Chenfeng Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, Danyang Zhang - 2023

2 papers in library cite

[38]AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

W. Zhong, R. Cui, Y. Guo, Yiqing Liang, S. Lu, Yuzhi Wang, A. Saied, Weizhu Chen, N. Duan - 2023

1 paper in library cites

[39]Chatglm3 Series: Open Bilingual Chat LLMS

C. Team - 2023

1 paper in library cites

[40]Cmath: Can Your Language Model Pass chinese elementary School Math Test?

T. Wei, J. Luan, Weizhou Liu, S. Dong, B. Wang - 2023

1 paper in library cites

[41]Draft, Sketch, and Prove: Guiding Formal Theorem Provers With Informal Proofs

A. Q. Jiang, S. Welleck, J. P. Zhou, Wentao Li, Joseph Liu, M. Jamnik, T. Lacroix, Yonghui Wu, G. Lample - 2022

1 paper in library cites

[42]Embracing Change and Resetting Expectations

T. Tao - 2023

1 paper in library cites

[43]Generative AI for Math: Part I - Mathpile: A Billion-Token-Scale Pretraining Corpus for Math

Zhengtao Wang, R. Xia, P. Liu - 2023

1 paper in library cites

[44]Glm: General Language Model Pretraining With Autoregressive Blank Infilling

Z. Du, Y. Qian, Xiaodong Liu, M. Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang - 2022

1 paper in library cites

[45]Hai-llm: 高效且轻量的大模型训练工具

H. Flyer - 2023

1 paper in library cites

[46]Inflection-2

I. Ai - 2023

1 paper in library cites

[47]LILA: A Unified Benchmark for Mathematical Reasoning

Swaroop Mishra, M. Finlayson, P. Lu, L. Tang, S. Welleck, Chitta Baral, T. Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, A. Kalyan - 2022

1 paper in library cites

[48]Minif2f: A Cross-System Benchmark for Formal Olympiad-Level Mathematics

K. Zheng, J. M. Han, S. Polu - 2021

1 paper in library cites

[49]Openwebmath: An Open Dataset of High-Quality Mathematical Web Text

K. Paster, M. D. Santos, Z. Azerbayev, Jimmy Lei Ba - 2023

1 paper in library cites

[50]PAL: Program-Aided Language Models

Leo Gao, Aman Madaan, Shuyan Zhou, U. Alon, P. Liu, Yining Yang, J. Callan, Graham Neubig - 2023

1 paper in library cites

[51]Preference Ranking Optimization for Human Alignment

Francis Song, B. Yu, M. Li, H. Yu, F. Huang, Yiwei Li, Haiming Wang - 2023

1 paper in library cites

[52]Redpajama: An Open Dataset for Training Large Language Models

T. Computer - 2023

1 paper in library cites

[53]Rrhf: Rank Responses to Align Language Models With Human Feedback Without Tears

Zhihao Yuan, H. Yuan, C. Tan, Wenyi Wang, S. Huang, F. Huang - 2023

1 paper in library cites

[54]Seallms - Large Language Models for southeast Asia

X. Nguyen, Wenxuan Zhang, Xiang Lisa Li, M. M. Aljunied, Q. Tan, L. C. Cheng, Guanduo Chen, Y. Deng, Shusheng Yang, C. L. Liu, Haowei Zhang, L. Bing - 2023

1 paper in library cites

[55]Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation

H. Xia, Tiezheng Ge, Peng Wang, S. Q. Chen, F. Wei, Zhifang Sui - 2023

1 paper in library cites

[56]The isabelle Framework

M. Wenzel, L. C. Paulson, T. Nipkow - 2008

1 paper in library cites

[57]Three Years of Experience With Sledgehammer, a Practical Link Between Automatic and Interactive Theorem Provers

L. C. Paulson - 2010

1 paper in library cites

[58]Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding

H. Xia, Zhilin Yang, Q. Dong, Peng Wang, Yiwei Li, Tiezheng Ge, T. Liu, Wentao Li, Zhifang Sui - 2024

1 paper in library cites

[59]Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

Collin Burns, P. Izmailov, J. H. Kirchner, Bowen Baker, Leo Gao, L. Aschenbrenner, Yanru Chen, A. Ecoffet, M. Joglekar, Jan Leike - 2023

1 paper in library cites

Cited by

3

papers in your library

Cites

23

papers in your library

Read

on June 1, 2026

Fun read and GRPO is a very nice idea!

Tags

RLHFVetto Study

Paper Aliases

No aliases