Papperoni

2024

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Carlos E. Jimenez, Jihan Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, K. R. Narasimhan

citations

Cite Score

AI summary

This paper introduces SWE-bench, a new benchmark with 2,294 real-world GitHub issues for evaluating language models in software engineering tasks, demonstrating that current state-of-the-art models (like Claude 2 and fine-tuned SWE-Llama) solve only the simplest issues (1.96% for Claude 2).

Main Contributions

Introduces SWE-bench, a challenging evaluation framework for language models consisting of 2,294 real-world software engineering problems from GitHub.
Proposes a task where language models edit codebases to resolve issues, requiring complex reasoning, environment interaction, and long context processing.
Develops SWE-Llama 7b and 13b, fine-tuned CodeLlama models, for repository-wide code edits.
Evaluates state-of-the-art models, revealing that even the best-performing (Claude 2) solves only 1.96% of issues, highlighting the difficulty of the benchmark.
Provides a training dataset, SWE-bench-train, with 19,000 non-testing task instances to foster open-model development.

Abstract

Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find real-world software engineering to be a rich, sustainable, and challenging testbed for evaluating the next generation of language models. To this end, we introduce SWE-bench, an evaluation framework consisting of 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue. Resolving issues in SWE-bench frequently requires understanding and coordinating changes across multiple functions, classes, and even files simultaneously, calling for models to interact with execution environments, process extremely long contexts and perform complex reasoning that goes far beyond traditional code generation tasks. Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. The best-performing model, Claude 2, is able to solve a mere 1.96% of the issues. Advances on SWE-bench represent steps towards LMs that are more practical, intelligent, and autonomous.

Citation Graph

Loading graph...

References [69]

Sort:

Filter:

[1]Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba - 2021

16 papers in library cite