Papperoni

2026

How Reliable Is Language Model Micro-Benchmarking?

Gregory Yauney, Shahzaib Saqib Warraich, Swabha Swayamdipta

citations

Cite Score

AI summary

This paper introduces Minimum Detectable Ability Difference (MDAD), a meta-evaluation measure to assess the reliability of language model micro-benchmarks by evaluating pairwise model rankings against full benchmarks, finding that random sampling is competitive at larger sizes (250+ examples) and that small micro-benchmarks often fail to distinguish models with similar performances.

Main Contributions

Introduces Minimum Detectable Ability Difference (MDAD) as a meta-evaluation measure for micro-benchmarking reliability.
Shows that no micro-benchmarking method consistently ranks model pairs that differ by 3.5 points (MMLU-Pro) or 4 points (BIG-bench Hard) of accuracy.
Demonstrates that 250 examples are often needed for consistent model ranking, at which point random sampling becomes competitive with other methods.
Reveals that over half of pairwise comparisons on MMLU-Pro micro-benchmarks with 25 examples are unlikely to be preserved for 8B instruction-tuned models.
Provides actionable guidance for micro-benchmark users and developers on the trade-off between evaluation efficiency and reliability.

Abstract

Micro-benchmarking offers a solution to the often prohibitive time and cost of language model development: evaluate on a very small subset of existing bench- marks. Can these micro-benchmarks, however, rank models as consistently as the full benchmarks they replace? And can they rank models more consistently than selecting a random subset of data points? In many scenarios, we find that the answer is no. We introduce a meta-evaluation measure for micro-benchmarking which investigates how well a micro-benchmark can rank two models as a function of their performance difference on the full benchmark. This approach can deter- mine which model pairs can be ranked correctly by a micro-benchmark, allowing for a finer-grained analysis of the trade-off between micro-benchmark size and reliability. Prior work has suggested selecting as few as 10 examples; we find that no micro-benchmarking method can consistently rank model pairs 3.5 points of accuracy apart on MMLU-Pro or 4 points apart on BIG-bench Hard. In order to consistently rank model pairs with relatively similar performances, we show that often as many as 250 examples must be selected, at which point random sampling is competitive with existing micro-benchmarking methods. When comparing only 8B instruction-tuned models on MMLU-Pro micro-benchmarks with 25 examples, we find that more than half of pairwise comparisons are not likely to be preserved. Our work provides actionable guidance for both micro-benchmark users and developers in navigating the trade-off between evaluation efficiency and reliability. Code and data are at: github.com/dill-lab/micro-benchmarking-reliability

Citation Graph

Loading graph...

References [46]

Sort:

Filter:

[1]Measuring Massive Multitask Language Understanding

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt - 2021

6 papers in library cite