Papperoni

2025

Deepseek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Deepseek Ai

citations

Cite Score

AI summary

This paper introduces DeepSeek-R1, an LLM trained with a novel reinforcement learning framework using Group Relative Policy Optimization (GRPO) that incentivizes emergent reasoning patterns like self-reflection and dynamic strategy adaptation, achieving superior performance on math, coding, and STEM tasks without human-labeled reasoning trajectories.

Main Contributions

Introduces DeepSeek-R1, an LLM whose reasoning abilities are incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories.
Proposes an RL framework that facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation.
The trained DeepSeek-R1 model achieves superior performance on verifiable tasks (mathematics, coding competitions, STEM fields) compared to models trained via conventional supervised learning.
Demonstrates that emergent reasoning patterns in large-scale models can guide and enhance reasoning capabilities in smaller models.
Achieves an average pass@1 score of 77.9% on AIME 2024, significantly outperforming human participants and other models.

Abstract

General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) (Brown et al., 2020; OpenAI, 2023) and chain-of-thought prompting (Wei et al., 2022b), have achieved considerable success on foundational reasoning tasks. However, this success is heavily contingent upon extensive human-annotated demonstrations, and models’ capabilities are still insufficient for more complex problems. Here we show that the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labeled reasoning trajectories. The proposed RL framework facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification, and dynamic strategy adaptation. Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions, and STEM fields, surpassing its counterparts trained via conventional supervised learning on human demonstrations. Moreover, the emergent reasoning patterns exhibited by these large-scale models can be systematically harnessed to guide and enhance the reasoning capabilities of smaller models.

Citation Graph

Loading graph...

References [90]

Sort:

Filter:

[1]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite