Papperoni

2024

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

Yuzhi Wang, Xueguang Ma, Ge Zhang, Yuan Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Zhejun Jiang, Tao Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, Weizhu Chen

Open PDF Google Scholar

citations

Cite Score

AI summary

This paper introduces MMLU-Pro, an enhanced dataset extending MMLU with more challenging, reasoning-focused questions and expanded choice sets (from 4 to 10 options), demonstrating increased difficulty (16-33% accuracy drop for models) and greater stability under varying prompts.

Main Contributions

Introduces MMLU-Pro, an enhanced benchmark with more challenging, reasoning-focused questions and expanded choice sets (from 4 to 10 options).
MMLU-Pro eliminates trivial and noisy questions found in the original MMLU dataset.
Shows that MMLU-Pro causes a significant drop in model accuracy (16% to 33%) compared to MMLU.
Demonstrates that MMLU-Pro provides greater stability under varying prompts, with sensitivity decreasing from 4-5% in MMLU to 2% in MMLU-Pro.
Highlights that models using Chain of Thought (CoT) reasoning perform better on MMLU-Pro, indicating more complex reasoning questions compared to the original MMLU.

Abstract

In the age of large-scale language models, benchmarks like the Massive Multitask Language Understanding (MMLU) have been pivotal in pushing the boundaries of what AI can achieve in language comprehension and reasoning across diverse domains. However, as models continue to improve, their performance on these benchmarks has begun to plateau, making it increasingly difficult to discern differences in model capabilities. This paper introduces MMLU-Pro, an enhanced dataset designed to extend the mostly knowledge-driven MMLU benchmark by integrating more challenging, reasoning-focused questions and expanding the choice set from four to ten options. Additionally, MMLU-Pro eliminates the trivial and noisy questions in MMLU. Our experimental results show that MMLU-Pro not only raises the challenge, causing a significant drop in accuracy by 16% to 33% compared to MMLU but also demonstrates greater stability under varying prompts. With 24 different prompt styles tested, the sensitivity of model scores to prompt variations decreased from 4-5% in MMLU to just 2% in MMLU-Pro. Additionally, we found that models utilizing Chain of Thought (CoT) reasoning achieved better performance on MMLU-Pro compared to direct answering, which is in stark con-trast to the findings on the original MMLU, indicating that MMLU-Pro includes more complex reasoning questions. Our assessments confirm that MMLU-Pro is a more discriminative benchmark to better track progress in the field.

Citation Graph

Loading graph...

References [48]

Sort:

Filter:

[1]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

25 papers in library cite