Papperoni

2021

What Will it Take to Fix Benchmarking in Natural Language Understanding?

Samuel R. Bowman, George E. Dahl

Open PDF Google Scholar

citations

Cite Score

14

AI summary

This paper argues that current NLU benchmarks are broken and proposes four criteria (validity, reliable annotation, statistical power, and disincentives for biased models) that future benchmarks should satisfy to facilitate progress in language understanding.

Main Contributions

Argues that current NLU evaluation benchmarks are broken due to saturation and biases.
Proposes four criteria for effective NLU benchmarks: validity, reliable annotation, adequate statistical power, and disincentives for biased models.
Discusses limitations of existing benchmark creation paradigms (naturally-occurring, expert-authored, crowdsourcing, adversarial filtering).
Outlines potential solutions for improving each criterion, including hybrid data collection and auxiliary bias evaluation metrics.
Emphasizes the need for community infrastructure and incentive design to address bias and improve evaluation.

Abstract

Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will perform poorly, but ultimately only obscures the abilities that we want our benchmarks to measure. In this position paper, we lay out four criteria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, the reliability with which they are annotated, their size, and the ways they handle social bias.

Citation Graph

Loading graph...

References [68]

Sort:

Filter:

[1]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[2]Language Models Are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei - 2020

21 papers in library cite

It's just training the GPT arch with more data and more params. Nothing too surprising, but kudos for identifying and formalizing few-shot learning.

[3]RoBERTa: A Robustly Optimized BERT Pretraining Approach

Yibo Liu, M. Ott, N. Goyal, J. Du, M. Joshi, Deli Chen, Omer Levy, Martha Lewis, Luke Zettlemoyer, Veselin Stoyanov - 2019

17 papers in library cite

I liked it a lot! It shows that you don't need to do something completely new to have good results and contribute to science. It could be a 5, but it's a 4 due to not bringing anything new

[4]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[5]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[6]Learning Word Vectors for Sentiment Analysis

A. L. Maas, R. E. Daly, P. T. Pham, Dong Huang, Andrew Y. Ng, Christopher Potts - 2011

12 papers in library cite

Very convoluted method. I did not like it and I had to go back twice to remember if I had read it or not. Main contribution is the IMDB movie review dataset. (It is cited by Mikolov as a comparison)

[7]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[8]Natural Questions: A Benchmark for Question Answering Research

T. Kwiatkowski, J. Palomaki, O. Rhinehart, Michael Collins, A. P. Parikh, C. Alberti, D. Epstein, Illia Polosukhin, M. Kelcey, Jacob Devlin, K. Lee, K. N. Toutanova, Llion Jones, M. W. Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, Slav Petrov - 2019

9 papers in library cite

The dataset and methodology is very nice - it's amazing to see how Google does the summaries in search. However, the paper is too complex with the math stuff - unnecessary.

[9]Know What You Don't Know: Un-Answerable Questions for SQuAD

P. Rajpurkar, R. Jia, Percy Liang - 2018

14 papers in library cite

It's alright... It's an extension to the other paper/dataset. I feel that it didn't need to be a full paper (maybe a 6-pager).

[10]Labeling Images With a Computer Game

Luis Von Ahn, Laura Dabbish - 2004

5 papers in library cite

It's alright. TBH I expected a bit more. It's a good idea but I feel that they were very optimist around people playing - and it didn't work

[11]SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2019

15 papers in library cite

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

[12]Adversarial Examples for Evaluating Reading Comprehension Systems

R. Jia, Percy Liang - 2017

11 papers in library cite

I liked it a lot! It's good to see people testing things rather than just trying to beat SotA!

[13]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[14]Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

R. T. Mccoy, Ellie Pavlick, Tal Linzen - 2019

5 papers in library cite

TBH I expected more. It's a very shallow analysis of the heuristics (but ok, props to them for making it simple). I think that ultimately they don't solve the problem.

[15]SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Yejin Choi - 2018

5 papers in library cite

Very nice methodology with the adversarial stuff! I think not too relevant though.

[16]The LAMBADA dataset: Word Prediction Requiring a Broad Discourse Context

D. Paperno, German Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, Raquel Fernandez - 2016

12 papers in library cite

Very nice paper - very interesting methodology to building it and very good when they bring a dataset that is meant to make machines to fail

[17]Utility Is in the Eye of the User: A Critique of NLP Leaderboards

Kawin Ethayarajh, Dan Jurafsky - 2020

3 papers in library cite

Ok, agreed. But no news and they don't provide a good solution to the problem.

[18]Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer

Colin Raffel, Noam Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, Wentao Li, P. J. Liu - 2019

17 papers in library cite

44 pages; T5 paper - "unifying framework where all text-based NLP problems are cast as text-to-text tasks"

Missing author list

[19]Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

2016

2 papers in library cite

[20]DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen - 2020

4 papers in library cite

"One of the final "classic" NLP architecture refinements that dominated leaderboards"

[21]Winogrande: An Adversarial Winograd Schema Challenge at Scale

K. Sakaguchi, R. L. Bras, C. Bhagavatula, Yejin Choi - 2019

4 papers in library cite

[22]Beyond Accuracy: Behavioral Testing of NLP Models With CheckList

M. T. Ribeiro, Tianhao Wu, C. Guestrin, Shivalika Singh - 2020

2 papers in library cite

[23]Lipstick on a Pig: Debiasing Methods Cover Up Systematic Gender Biases in Word Embeddings but Do Not Remove Them

H. Gonen, Y. Goldberg - 2019

4 papers in library cite

Debiasing doesnt work as expected!

[24]Stress Test Evaluation for Natural Language Inference

A. Naik, A. Ravichander, N. M. Sadeh, C. P. Rose, Graham Neubig - 2018

4 papers in library cite

A bit flopped but I like the idea - they test if the system is truly making inferences.

[25]With Little Power Comes Great Responsibility

D. Card, P. Henderson, U. Khandelwal, R. Jia, K. Mahowald, Dan Jurafsky - 2020

2 papers in library cite

[26]Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark

Nikita Nangia, Samuel R. Bowman - 2019

3 papers in library cite

I like the premise but seems a bit flopped

[27]Gender Bias in Coreference Resolution

R. Rudinger, J. Naradowsky, B. Leonard, B. V. Durme - 2018

6 papers in library cite

[28]Social Bias in Elicited Natural Language Inferences

R. Rudinger, C. May, B. V. Durme - 2017

3 papers in library cite

[29]To Test Machine Comprehension, Start by Defining Comprehension

J. Dunietz, G. Burnham, A. Bharadwaj, O. Rambow, J. C. Carroll, D. Ferrucci - 2020

1 paper in library cites

[30]Evaluating Machines by Their Real-World Language Use

Rowan Zellers, Ari Holtzman, E. Clark, Lianhui Qin, Ali Farhadi, Yejin Choi - 2020

2 papers in library cite

[31]A Survey of 25 Years of Evaluation

K. W. Church, J. Hestness - 2019

1 paper in library cites

[32]Benchmarking Machine Reading Comprehension: A Psychological Perspective

S. Sugawara, P. Stenetorp, A. Aizawa - 2020

1 paper in library cites

[33]Language (Technology) Is Power: A Critical Survey of "Bias" in NLP

S. L. Blodgett, S. Barocas, H. D. Iii, H. Wallach - 2020

7 papers in library cite

[34]Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation

A. Poliak, A. Haldar, R. Rudinger, J. E. Hu, Ellie Pavlick, A. S. White, B. V. Durme - 2018

4 papers in library cite

[35]Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science

E. M. Bender, B. Friedman - 2018

4 papers in library cite

[36]Adversarial Nli: A New Benchmark for Natural Language Understanding

Y. Nie, A. Williams, E. Dinan, Mohit Bansal, Jason Weston, Douwe Kiela - 2019

3 papers in library cite

[37]Climbing Towards NLU: On Meaning, Form, and Understanding in the Age of Data

E. M. Bender, A. Koller - 2020

3 papers in library cite

[38]Datasheets for Datasets

T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, K. Crawford - 2018

3 papers in library cite

[39]Experience Grounds Language

Yonatan Bisk, Ari Holtzman, J. Thomason, Jacob Andreas, Yoshua Bengio, J. Chai, Mirella Lapata, A. Lazaridou, J. May, A. Nisnevich - 2020

3 papers in library cite

[40]Using the Framework

R. Cooper, D. Crouch, J. Eijck, C. Fox, J. Genabith, J. Jaspars, H. Kamp, D. Milward, M. Pinkal, M. Poesio, S. Pulman, T. Briscoe, H. Maier, K. Konrad - 1996

3 papers in library cite

[41]Cosmos qa: Machine Reading Comprehension With Contextual Commonsense Reasoning

L. Huang, R. L. Bras, C. Bhagavatula, Yejin Choi - 2019

2 papers in library cite

[42]Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns

K. Webster, M. Recasens, V. Axelrod, J. Baldridge - 2018

2 papers in library cite

[43]ORB: An Open Reading Benchmark for Comprehensive Evaluation of Machine Reading Comprehension

D. Dua, A. Gottumukkala, A. Talmor, Shivalika Singh, Matt Gardner - 2019

2 papers in library cite

[44]Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment

M. Tsuchiya - 2018

2 papers in library cite

[45]Probing Neural Network Comprehension of Natural Language Arguments

T. Niven, H. Y. Kao - 2019

2 papers in library cite

[46]Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task

A. Ettinger, S. Rao, H. D. Iii, E. Bender - 2017

2 papers in library cite

[47]A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation

M. Poesio, J. Chamberlain, S. Paun, J. Yu, A. Uma, U. Kruschwitz - 2019

1 paper in library cites

[48]Adversarial Filters of Dataset Biases

R. L. Bras, Swabha Swayamdipta, C. Bhagavatula, Rowan Zellers, M. E. Peters, Ashish Sabharwal, Yejin Choi - 2020

1 paper in library cites

[49]Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options

C. Vania, R. Chen, Samuel R. Bowman - 2020

1 paper in library cites

[50]Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets

S. Sugawara, P. Stenetorp, K. Inui, A. Aizawa - 2020

1 paper in library cites

[51]Creating Zombilingo, a Game With a Purpose for Dependency Syntax Annotation

K. Fort, B. Guillaume, H. Chastant - 2014

1 paper in library cites

[52]Evaluating Models' Local Decision Boundaries via Contrast Sets

Matt Gardner, Y. Artzi, V. Basmov, Jonathan Berant, B. Bogin, S. Chen, P. Dasigi, D. Dua, Y. Elazar, A. Gottumukkala, N. Gupta, Hananneh Hajishirzi, G. Ilharco, Daniel Khashabi, K. Lin, Joseph Liu, N. F. Liu, P. Mulcaire, Q. Ning, Shivalika Singh, Noah A. Smith, S. Subramanian, R. Tsarfaty, E. Wallace, A. Zhang, B. Zhou - 2020

1 paper in library cites

[53]Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems

Svetlana Kiritchenko, Saif M. Mohammad - 2018

1 paper in library cites

[54]India’s Engineers Have Thrived in Silicon valley. So Has Its Caste System

N. Tiku - 2020

1 paper in library cites

[55]Inherent Disagreements in Human Textual Inferences

Ellie Pavlick, T. Kwiatkowski - 2019

1 paper in library cites

[56]Intermediate-Task Transfer Learning With Pretrained Language Models: When and Why Does It Work?

Y. Pruksachatkun, Jason Phang, Haozhe Liu, Phu Mon Htut, X. Zhang, R. Y. Pang, C. Vania, K. Kann, Samuel R. Bowman - 2020

1 paper in library cites

[57]Is a Burrito a Sandwich? Exploring Race, Class, and Culture in Contracts

M. Florestall - 2008

1 paper in library cites

[58]It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners

Timo Schick, Hinrich Schutze - 2020

1 paper in library cites

[59]Large-Scale Pretraining for Neural Machine Translation With Tens of Billions of Sentence Pairs

Y. Meng, Xiang Ren, Z. Sun, Xiang Lisa Li, A. Yuan, F. Wu, Jeffrey Li - 2019

1 paper in library cites

[60]Metrology for AI: From Benchmarks to Instruments

C. Welty, P. Paritosh, L. Aroyo - 2019

1 paper in library cites

[61]New Protocols and Negative Results for Textual Entailment Data Collection

Samuel R. Bowman, J. Palomaki, L. B. Soares, E. Pitler - 2020

1 paper in library cites

[62]OCNLI: Original Chinese Natural Language Inference

Han Hu, Kyle Richardson, L. Xu, Lei Li, S. Kubler, L. Moss - 2020

1 paper in library cites

[63]OpenAI Launches an API to Commercialize Its Research

K. Wiggers - 2020

1 paper in library cites

[64]The Gamification of Work: Lessons From Crowdsourcing

B. Morschheuser, J. Hamari - 2019

1 paper in library cites

[65]Towards Accountability for Machine Learning Datasets: Practices From Software Engineering and Infrastructure

B. Hutchinson, A. Smart, A. Hanna, E. Denton, C. Greer, O. Kjartansson, P. Barnes, M. Mitchell - 2021

1 paper in library cites

[66]UNQOVERing Stereotyping Biases via Underspecified Questions

Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Vivek Srikumar - 2020

1 paper in library cites

[67]What Does BERT Learn From Multiple-Choice Reading Comprehension Datasets?

C. Si, Shijie Wang, M. Y. Kan, J. J. Jiang - 2019

1 paper in library cites

[68]What Question Answering Can Learn From Trivia Nerds

J. B. Graber, B. Borschinger - 2020

1 paper in library cites

Cited by

1

papers in your library

Cites

28

papers in your library

Read

on June 3, 2026

It is a nice push but really there are no clear outcomes. The proposal is very shallow.

Tags

Benchmark

Paper Aliases

No aliases