Papperoni

2024

LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

Naman Jain, King Han, Albert Gu, Wentao Li, Fanjia Yan, Tong Zhang, Shijie Wang, Armando Solar Lezama, Koushik Sen, Ion Stoica

Open PDF Google Scholar

citations

Cite Score

AI summary

This paper introduces LIVECODEBENCH, a new comprehensive and contamination-free benchmark for evaluating Large Language Models for code, collecting over 500 new problems from LEETCODE, ATCODER, and CODEFORCES, and evaluating models across scenarios like code generation, self-repair, code execution, and test output prediction, revealing contamination and overfitting issues in existing benchmarks.

Main Contributions

Introduction of LIVECODEBENCH, a new benchmark for holistic and contamination-free evaluation of LLMs for code.
Collection of new problems over time from LEETCODE, ATCODER, and CODEFORCES to ensure contamination-free evaluation.
Evaluation across a broader range of code-related capabilities, including self-repair, code execution, and test output prediction.
Empirical findings on contamination in existing benchmarks (e.g., DEEPSEEK, GPT-4-O) and potential overfitting in HUMANEVAL.
Release of all prompts, model completions, and a toolkit for adding new scenarios and models to foster community analysis.

Abstract

Large Language Models (LLMS) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMS are developed, existing evaluation benchmarks (e.g., HUMANEVAL, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LIVECODEBENCH, a comprehensive and contamination-free evaluation of LLMS for code, which collects new problems over time from contests across three competition platforms, namely LEETCODE, ATCODER, and CODEFORCES. Notably, our benchmark also focuses on a broader range of code-related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LIVECODEBENCH hosts over five hundred coding problems that were published between May 2023 and May 2024. We have evaluated 18 base LLMS and 34 instruction-tuned LLMS on LIVECODEBENCH. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and models.

Citation Graph

Loading graph...

References [95]

Sort:

Filter:

[1]Evaluating Large Language Models Trained on Code

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, Wojciech Zaremba - 2021

16 papers in library cite