Papperoni

2020

Utility Is in the Eye of the User: A Critique of NLP Leaderboards

Kawin Ethayarajh, Dan Jurafsky

Open PDF Google Scholar

citations

Cite Score

13

AI summary

This paper critiques the current NLP leaderboard paradigm, arguing that its focus on performance-based evaluation neglects other important model qualities like fairness and efficiency, and proposes increased transparency by reporting practical statistics (e.g., model size, energy efficiency) to better align with user utility.

Main Contributions

Critiques current NLP leaderboards for prioritizing performance over other valuable qualities like fairness, compactness, and energy efficiency.
Frames leaderboards and NLP practitioners as consumers of models, using microeconomic theory to analyze the divergence in their utility functions.
Identifies limitations in leaderboard design, including non-smooth utility for leaderboards and the neglect of prediction costs (e.g., model size, energy efficiency, latency).
Advocates for increased transparency on leaderboards, recommending the reporting of practical statistics for models.
Proposes a dynamic, customizable leaderboard interface allowing users to re-rank models based on their individual utility functions and preferences.

Abstract

Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models. While this leaderboard paradigm has been remarkably successful, a historical focus on performance-based evaluation has been at the expense of other qualities that the NLP community values in models, such as compactness, fairness, and energy efficiency. In this opinion paper, we study the divergence between what is incentivized by leaderboards and what is useful in practice through the lens of microeconomic theory. We frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. With this framing, we formalize how leaderboards in their current form can be poor proxies for the NLP community at large. For example, a highly inefficient model would provide less utility to practitioners but not to a leaderboard, since it is a cost that only the former must bear. To allow practitioners to better estimate a model's utility to them, we advocate for more transparency on leaderboards, such as the reporting of statistics that are of practical concern (e.g., model size, energy efficiency, and inference latency).

Citation Graph

Loading graph...

References [56]

Sort:

Filter:

[1]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[2]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[3]DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter

Thomas Wolf - 2019

6 papers in library cite

Surprisingly simple and amazing results!

[4]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[5]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[6]Energy and Policy Considerations for Deep Learning in NLP

E. Strubell, A. Ganesh, Andrew Mccallum - 2019

3 papers in library cite

Very nice push and maybe the first one to actually raise a flag - also, very contemporary and relevant discussion

[7]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[8]Know What You Don't Know: Un-Answerable Questions for SQuAD

P. Rajpurkar, R. Jia, Percy Liang - 2018

14 papers in library cite

It's alright... It's an extension to the other paper/dataset. I feel that it didn't need to be a full paper (maybe a 6-pager).

[9]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

Oren Etzioni - 2019

4 papers in library cite

I like the push a lot! Specially when it comes to make research more democratic, which I personally benefit from. Also, AllenAI seems like they do good, honest work!

[11]Adversarial Examples for Evaluating Reading Comprehension Systems

R. Jia, Percy Liang - 2017

11 papers in library cite

I liked it a lot! It's good to see people testing things rather than just trying to beat SotA!

[12]The Natural Language Decathlon: Multitask Learning as Question Answering

Richard Socher - 2018

9 papers in library cite

Very nice results, but TBH the architecture is waaaay to complicated

[13]Model Cards for Model Reporting

M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, T. Gebru - 2018

5 papers in library cite

Seems like something foundational - we use at nubank!

[14]A Simple but Tough-to-Beat Baseline for Sentence Embeddings

S. Arora, Yiqing Liang, T. Ma - 2017

4 papers in library cite

[15]The Ladder: A Reliable Leaderboard for Machine Learning Competitions

A. Blum, Moritz Hardt - 2015

2 papers in library cite

Seems like a better leaderboard

[16]Gender Bias in Coreference Resolution

R. Rudinger, J. Naradowsky, B. Leonard, B. V. Durme - 2018

6 papers in library cite

[17]Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods

J. Zhao, Tianle Wang, M. Yatskar, V. Ordonez, K. W. Chang - 2018

3 papers in library cite

[18]Language (Technology) Is Power: A Critical Survey of "Bias" in NLP

S. L. Blodgett, S. Barocas, H. D. Iii, H. Wallach - 2020

7 papers in library cite

[19]Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science

E. M. Bender, B. Friedman - 2018

4 papers in library cite

[20]SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

E. Agirre, C. Banea, C. Cardie, D. M. Cer, M. T. Diab, A. G. Agirre, W. Guo, R. Mihalcea, G. Rigau, J. Wiebe - 2014

4 papers in library cite

[21]Stereoset: Measuring Stereotypical Bias in Pretrained Language Models

Moin Nadeem, A. Bethke, Siva Reddy - 2020

4 papers in library cite

[22]Adversarial Nli: A New Benchmark for Natural Language Understanding

Y. Nie, A. Williams, E. Dinan, Mohit Bansal, Jason Weston, Douwe Kiela - 2019

3 papers in library cite

[23]Datasheets for Datasets

T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, K. Crawford - 2018

3 papers in library cite

[24]How Can We Accelerate Progress Towards Human-Like Linguistic Generalization?

Tal Linzen - 2020

3 papers in library cite

[25]Show Your Work: Improved Reporting of Experimental Results

J. Dodge, Suchin Gururangan, D. Card, Richard Schwartz, Noah A. Smith - 2019

3 papers in library cite

[26]ELECTRA: Pre-Training Text Encoders as Discriminators Rather Than Generators

K. Clark, M. T. Luong, Quoc V. Le, Christopher D. Manning - 2020

2 papers in library cite

[27]Identifying and Reducing Gender Bias in Word-Level Language Models

S. Bordia, S. Bowman - 2019

2 papers in library cite

[28]The Effect of Natural Distribution Shift on Question Answering Models

John Miller, Karl Krauth, Benjamin Recht, Ludwig Schmidt - 2020

2 papers in library cite

[29]Adversarial Attacks on Deep-Learning Models in Natural Language Processing: A Survey

W. E. Zhang, Q. Z. Sheng, A. Alhazmi, Chun-Liang Li - 2020

1 paper in library cites

[30]Black Is to Criminal as caucasian Is to Police: Detecting and Removing Multi-Class Bias in Word Embeddings

T. Manzini, L. Y. Chong, A. W. Black, Y. Tsvetkov - 2019

1 paper in library cites

[31]Certified Defenses Against Adversarial Examples

A. Raghunathan, Jacob Steinhardt, Percy Liang - 2018

1 paper in library cites

[32]Certified Robustness to Adversarial Word Substitutions

R. Jia, A. Raghunathan, K. Goksel, Percy Liang - 2019

1 paper in library cites

[33]Climbing a Shaky Ladder: Better Adaptive Risk Estimation

Moritz Hardt - 2017

1 paper in library cites

[34]Consumption Theory in Terms of Revealed Preference

P. A. Samuelson - 1948

1 paper in library cites

[35]Distributionally Robust Language Modeling

Y. Oren, S. Sagawa, Tatsunori Hashimoto, Percy Liang - 2019

1 paper in library cites

[36]Dynabert: Dynamic BERT With Adaptive Width and Depth

L. Hou, L. Shang, Xu Jiang, Qian Liu - 2020

1 paper in library cites

[37]Equality of Opportunity in Supervised Learning

Moritz Hardt, E. Price, N. Srebro - 2016

1 paper in library cites

[38]Fairness in Machine Learning

S. Barocas, Moritz Hardt, A. Narayanan - 2017

1 paper in library cites

[39]Fairness Without Demographics in Repeated Loss Minimization

Tatsunori Hashimoto, M. Srivastava, H. Namkoong, Percy Liang - 2018

1 paper in library cites

[40]Gobo: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference

A. H. Zadeh, A. Moshovos - 2020

1 paper in library cites

[41]How the Transformers Broke NLP Leaderboards

A. Rogers - 2019

1 paper in library cites

[42]Is Your Classifier Actually Biased? Measuring Fairness Under Uncertainty With Bernstein Bounds

Kawin Ethayarajh - 2020

1 paper in library cites

[43]Justice as Fairness: A Restatement

J. Rawls - 2001

1 paper in library cites

[44]Ladabert: Lightweight Adaptation of BERT Through Hybrid Model Compression

Y. Mao, Yuzhi Wang, Chiyu Wu, Chiyuan Zhang, Yuzhi Wang, Yining Yang, Q. Zhang, Y. Tong, Jinze Bai - 2020

1 paper in library cites

[45]Overview of Results of the MUC-6 Evaluation

B. Sundheim - 1995

1 paper in library cites

[46]Part 5: Machine Translation Evaluation

B. Dorr - 2011

1 paper in library cites

[47]Peer Review in NLP: Reject-if-Not-Sota

A. Rogers - 2020

1 paper in library cites

[48]Principles of Economics

N. G. Mankiw - 2020

1 paper in library cites

[49]Questionable Answers in Question Answering Research: Reproducibility and Variability of Published Results

M. Crane - 2018

1 paper in library cites

[50]Rotate King to Get Queen: Word Relationships as Orthogonal Transformations in Embedding Space

Kawin Ethayarajh - 2019

1 paper in library cites

[51]Sem 2013 Shared Task: Semantic Textual Similarity, Including a Pilot on Typed-Similarity

E. Agirre, D. Cer, M. Diab, A. G. Agirre, W. Guo - 2013

1 paper in library cites

[52]Semeval-2015 Task 2: Semantic Textual Similarity, English, Spanish and Pilot on Interpretability

E. Agirre, C. Banea, C. Cardie, D. M. Cer, M. T. Diab, A. G. Agirre, W. Guo, I. L. Gazpio, M. Maritxalar, R. Mihalcea - 2015

1 paper in library cites

[53]Semidefinite Relaxations for Certifying Robustness to Adversarial Examples

A. Raghunathan, Jacob Steinhardt, P. S. Liang - 2018

1 paper in library cites

[54]Towards Understanding Linear Word Analogies

Kawin Ethayarajh, David Duvenaud, G. Hirst - 2019

1 paper in library cites

[55]Understanding Undesirable Word Embedding Associations

Kawin Ethayarajh, David Duvenaud, G. Hirst - 2019

1 paper in library cites

[56]Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline

Kawin Ethayarajh - 2018

1 paper in library cites

Cited by

3

papers in your library

Cites

17

papers in your library

Read

on June 2, 2026

Ok, agreed. But no news and they don't provide a good solution to the problem.

Tags

Paper Aliases

No aliases