Papperoni

2024

LiveBench: A Challenging, Contamination-Limited LLM Benchmark

Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Shantanu Jain, Ravid Shwartz Ziv, Naman Jain, Khalid Saifullah, Sreemanti Dey, Shubh Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann Lecun, Tom Goldstein, Willie Neiswanger, Micah Goldblum

Open PDF Google Scholar

citations

Cite Score

AI summary

This paper introduces LiveBench, a new LLM benchmark designed to resist test set contamination and mitigate biases from human or LLM judges, featuring frequently-updated questions from recent information sources, automatic objective ground-truth scoring, and a diverse set of challenging tasks across math, coding, reasoning, language, instruction following, and data analysis.

Main Contributions

Introduces LiveBench, a novel LLM benchmark designed to be resistant to test set contamination and biases from LLM/human judging.
LiveBench features frequently-updated questions from recent information sources (e.g., math competitions, arXiv papers, news articles, datasets).
Employs automatic scoring based on objective ground-truth values, eliminating the need for LLM or human judges.
Includes a wide variety of challenging tasks spanning math, coding, reasoning, language, instruction following, and data analysis.
Demonstrates that LiveBench is difficult for current models (top models achieve below 70% accuracy) and is designed for continuous updates to remain challenging.

Abstract

Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In this work, we introduce a new benchmark for LLMs designed to be resistant to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release LiveBench, the first benchmark that (1) contains frequently-updated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-limited versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size. LiveBench is difficult, with top models achieving below 70% accuracy. We release all questions, code, and model answers, Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time so that LiveBench can distinguish between the capabilities of LLMs as they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models.

Citation Graph

Loading graph...

References [77]

Sort:

Filter:

[1]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

40 papers in library cite