Papperoni

2019

SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

A. Wang, Y. Pruksachatkun, Nikita Nangia, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman

Open PDF Google Scholar

citations

Cite Score

63

AI summary

This paper introduces SuperGLUE, a new benchmark styled after GLUE with more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE consists of a public leaderboard built around eight language understanding tasks, drawing on existing data, accompanied by a single-number performance metric, and an analysis toolkit.

Main Contributions

Introduces SuperGLUE benchmark to evaluate general-purpose language understanding systems.
SuperGLUE includes more challenging and diverse tasks, comprehensive human baselines, and improved code support compared to GLUE.
SuperGLUE retains the two hardest tasks in GLUE.
SuperGLUE expands the set of task formats to include coreference resolution and question answering (QA).
Provides a modular toolkit for work on pretraining, multi-task learning, and transfer learning in NLP, built around standard tools including PyTorch and AllenNLP.

Abstract

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

Citation Graph

Loading graph...

References [73]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding

Jacob Devlin, M. W. Chang, K. Lee, Kristina Toutanova - 2018

39 papers in library cite

Simply amazing. It's very impressive how they make a leap vs. existing stuff (you can see from the references, pretty much no one is doing what they are doing, other than GPT)

[3]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[4]Distilling the Knowledge in a Neural Network

Jeffrey Dean - 2015

6 papers in library cite

Very nice, very relevant.

[5]WordNet: A Lexical Database for English

G. Miller - 1995

5 papers in library cite

Meh. It seems like it was publish so that people could cite the dataset somehow. Nothing interesting, but quick read and very used.

[6]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[7]Automatic Differentiation in PyTorch

A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Devito, Zongyu Lin, A. Desmaison, L. Antiga, Adam Lerer - 2017

3 papers in library cite

Very boring and poorly explained. It looks like they said "we need to cash in some citations! quick, publish anything you have!!!"

[8]Improving Language Understanding by Generative Pre-Training

Alec Radford, K. Narasimhan, T. Salimans, Ilya Sutskever - 2018

23 papers in library cite

Very simple and very nice! Easy to understand and revolutionary maybe?

[9]XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Z. Dai, Yining Yang, J. Carbonell, Ruslan Salakhutdinov, Quoc V. Le - 2019

11 papers in library cite

The method is nice and the results are very good but this paper is just soooo hard to follow...

[10]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[11]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[12]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[13]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[14]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[15]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[16]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[17]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[18]Automatically Constructing a Corpus of Sentential Paraphrases

W. Dolan, Chris Brockett - 2005

9 papers in library cite

Small dataset, questionable methodology, not useful for training models

[19]Adversarial Examples for Evaluating Reading Comprehension Systems

R. Jia, Percy Liang - 2017

11 papers in library cite

I liked it a lot! It's good to see people testing things rather than just trying to beat SotA!

[20]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[21]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[22]Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

R. T. Mccoy, Ellie Pavlick, Tal Linzen - 2019

5 papers in library cite

TBH I expected more. It's a very shallow analysis of the heuristics (but ok, props to them for making it simple). I think that ultimately they don't solve the problem.

[23]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[24]SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Yejin Choi - 2018

5 papers in library cite

Very nice methodology with the adversarial stuff! I think not too relevant though.

[25]The Natural Language Decathlon: Multitask Learning as Question Answering

Richard Socher - 2018

9 papers in library cite

Very nice results, but TBH the architecture is waaaay to complicated

[26]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[27]BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

C. Clark, K. Lee, M. W. Chang, T. Kwiatkowski, Michael Collins, Kristina Toutanova - 2019

4 papers in library cite

Part of SuperGLUE

[28]Neural Network Acceptability Judgments

Alex Warstadt, A. Singh, S. Bowman - 2018

8 papers in library cite

CoLA dataset

[29]How Not to Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation

C. L. Liu, Ryan Lowe, I. Serban, M. Noseworthy, L. Charlin, J. Pineau - 2016

3 papers in library cite

They discuss how to game metrics

[30]Multi-Task Deep Neural Networks for Natural Language Understanding

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao - 2019

6 papers in library cite

[31]Allennlp: A Deep Semantic Natural Language Processing Platform

Matt Gardner, J. Grus, M. Neumann, Oyvind Tafjord, P. Dasigi, N. Liu, M. Peters, M. Schmitz, Luke Zettlemoyer - 2018

5 papers in library cite

Framework from alenai

[32]Born Again Neural Networks

T. Furlanello, Zachary C. Lipton, M. Tschannen, L. Itti, A. Anandkumar - 2018

1 paper in library cites

They use knowledge distillation - sounds relevant.

[33]Quac: Question Answering in Context

E. Choi, He He, M. Iyyer, M. Yatskar, W. T. Yih, Yejin Choi, Percy Liang, Luke Zettlemoyer - 2018

8 papers in library cite

[34]Lipstick on a Pig: Debiasing Methods Cover Up Systematic Gender Biases in Word Embeddings but Do Not Remove Them

H. Gonen, Y. Goldberg - 2019

4 papers in library cite

Debiasing doesnt work as expected!

[35]Senteval: An Evaluation Toolkit for Universal Sentence Representations

Alexis Conneau, Douwe Kiela - 2018

5 papers in library cite

Sentence embedding eval toolkit?

[36]Stress Test Evaluation for Natural Language Inference

A. Naik, A. Ravichander, N. M. Sadeh, C. P. Rose, Graham Neubig - 2018

4 papers in library cite

A bit flopped but I like the idea - they test if the system is truly making inferences.

[37]Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark

Nikita Nangia, Samuel R. Bowman - 2019

3 papers in library cite

I like the premise but seems a bit flopped

[38]Gender Bias in Coreference Resolution

R. Rudinger, J. Naradowsky, B. Leonard, B. V. Durme - 2018

6 papers in library cite

[39]What Do You Learn From Context? Probing for Sentence Structure in Contextualized Word Representations

I. Tenney, P. Xia, Berlin Chen, A. Wang, A. Poliak, R. T. Mccoy, N. Kim, B. V. Durme, S. Bowman, Dipanjan Das, Ellie Pavlick - 2019

4 papers in library cite

Analysis of how transformers learn phrase structure

[40]Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods

J. Zhao, Tianle Wang, M. Yatskar, V. Ordonez, K. W. Chang - 2018

3 papers in library cite

[41]The Fifth PASCAL Recognizing Textual Entailment Challenge

L. Bentivogli, Peter Clark, Ido Dagan, D. Giampiccolo - 2009

7 papers in library cite

[42]The Third PASCAL Recognizing Textual Entailment Challenge

D. Giampiccolo, Bernardo Magnini, Ido Dagan, B. Dolan - 2007

7 papers in library cite

[43]SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

D. Cer, M. Diab, E. Agirre, I. L. Gazpio, L. Specia - 2017

6 papers in library cite

[44]The Second PASCAL Recognising Textual Entailment Challenge

R. B. Haim, Ido Dagan, B. Dolan, L. Ferro, D. Giampiccolo, Bernardo Magnini, I. Szpektor - 2006

6 papers in library cite

[45]A Surprisingly Robust Trick for Winograd Schema Challenge

V. Kocijan, A. M. Cretu, O. M. Camburu, Y. Yordanov, T. Lukasiewicz - 2019

4 papers in library cite

[46]Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning

M. Roemmele, C. A. Bejan, A. S. Gordon - 2011

4 papers in library cite

[47]Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation

A. Poliak, A. Haldar, R. Rudinger, J. E. Hu, Ellie Pavlick, A. S. White, B. V. Durme - 2018

4 papers in library cite

[48]Looking Beyond the Surface: A Challenge Set for Reading Comprehension Over Multiple Sentences

Daniel Khashabi, S. Chaturvedi, M. Roth, Shyam Upadhyay, Dan Roth - 2018

4 papers in library cite

[49]ReCoRD: Bridging the Gap Between Human and Machine Commonsense Reading Comprehension

S. Zhang, Xiaodong Liu, Joseph Liu, Jianfeng Gao, K. Duh, B. V. Durme - 2018

4 papers in library cite

[50]BAM! Born-Again Multi-Task Networks for Natural Language Understanding

K. Clark, M. Luong, U. Khandelwal, C. Manning, Quoc Le - 2019

3 papers in library cite

[51]End-to-End Neural Coreference Resolution

K. Lee, Luheng He, Martha Lewis, L. S. Zettlemoyer - 2017

3 papers in library cite

[52]Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding

Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao - 2019

3 papers in library cite

[53]Jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models

Y. Pruksachatkun, P. Yeres, Haozhe Liu, Jason Phang, Phu Mon Htut, A. Wang, I. Tenney, Samuel R. Bowman - 2020

3 papers in library cite

[54]Non-Entailed Subsequences as a Challenge for Natural Language Inference

R. T. Mccoy, Tal Linzen - 2019

3 papers in library cite

[55]The CommitmentBank: Investigating Projection in Naturally Occurring Discourse

M. C. D. Marneffe, M. Simons, J. Tonhauser - 2019

3 papers in library cite

[56]Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns

K. Webster, M. Recasens, V. Axelrod, J. Baldridge - 2018

2 papers in library cite

[57]Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-Data Tasks

Jason Phang, T. Fevry, Samuel R. Bowman - 2018

2 papers in library cite

[58]SocialIQa: Common-Sense Reasoning About Social Interactions

Maarten Sap, Hannah Rashkin, Deli Chen, Ronan Lebras, Yejin Choi - 2019

2 papers in library cite

[59]A Corpus and Model Integrating Multiword Expressions and Supersenses

N. Schneider, Noah A. Smith - 2015

1 paper in library cites

[60]Gender Bias in Neural Natural Language Processing

K. Lu, P. Mardziel, F. Wu, P. Amancharla, A. Datta - 2018

1 paper in library cites

[61]Have You Lost the Thread? Discovering Ongoing Conversations in Scattered Dialog Blocks

F. M. Zanzotto, L. Ferrone - 2017

1 paper in library cites

[62]Identifying Well-Formed Natural Language Questions

Manaal Faruqui, Dipanjan Das - 2018

1 paper in library cites

[63]Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets

N. F. Liu, Richard Schwartz, Noah A. Smith - 2019

1 paper in library cites

[64]Linguistic Knowledge and Transferability of Contextual Representations

N. F. Liu, Matt Gardner, Yonatan Belinkov, M. E. Peters, Noah A. Smith - 2019

1 paper in library cites

[65]Modeling Empathy and Distress in Reaction to News Stories

S. Buechel, A. Buffone, B. Slaff, L. Ungar, João Sedoc - 2018

1 paper in library cites

[66]Multilingual Constituency Parsing With Self-Attention and Pre-Training

N. Kitaev, S. Cao, Dan Klein - 2019

1 paper in library cites

[67]PAWS: Paraphrase Adversaries From Word Scrambling

Y. Z. Zhang, J. Baldridge, Luheng He - 2019

1 paper in library cites

[68]Re-Evaluation the Role of Bleu in Machine Translation Research

Chris Callison Burch, M. Osborne, P. Koehn - 2006

1 paper in library cites

[69]Repurposing Entailment for Multi-Hop Question Answering Tasks

Harsh Trivedi, H. Kwon, Tushar Khot, Ashish Sabharwal, N. Balasubramanian - 2019

1 paper in library cites

[70]Snorkel drybell: A Case Study in Deploying Weak Supervision at Industrial Scale

S. H. Bach, D. Rodriguez, Yibo Liu, C. Luo, H. Shao, C. Xia, S. Sen, A. Ratner, B. Hancock, H. Alborzi, R. Kuchhal, C. Re, R. Malkin - 2018

1 paper in library cites

[71]Ultra-Fine Entity Typing

E. Choi, Omer Levy, Yejin Choi, Luke Zettlemoyer - 2018

1 paper in library cites

[72]Verbnet: A Broad-Coverage, Comprehensive Verb Lexicon

K. K. Schuler - 2005

1 paper in library cites

[73]WiC: The Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations

M. T. Pilehvar, J. C. Collados - 2019

1 paper in library cites

Cited by

15

papers in your library

Cites

40

papers in your library

Read

on December 28, 2025

Nothing too surprising, just getting a bunch of stuff from different places and putting it all together. At least they do a good analysis of the benchmark vs. existing methodologies.

Tags

Paper Aliases

No aliases