Papperoni

2018

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman

Open PDF Google Scholar

citations

Cite Score

82

AI summary

The paper introduces GLUE, a multi-task benchmark for NLU, comprising diverse tasks, limited data, and a diagnostic test suite. Baselines using ELMO achieve improved performance via multi-task training, yet reveal the necessity for enhanced general NLU systems.

Main Contributions

Introduces GLUE benchmark for evaluating NLU models across diverse tasks.
Includes tasks with limited training data to encourage models sharing general linguistic knowledge.
Provides a hand-crafted diagnostic test suite for detailed linguistic analysis of models.
Evaluates baselines using transfer and representation learning methods.
Finds that multi-task training on all tasks performs better than training a separate model per task.

Abstract

For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems.

Citation Graph

Loading graph...

References [52]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin - 2017

47 papers in library cite

I mean... it introduced Transformers!

[3]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[4]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[5]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[6]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[7]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[8]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[9]SQuAD: 100,000+ Questions for Machine Comprehension of Text

P. Rajpurkar, J. Zhang, K. Lopyrev, Percy Liang - 2016

37 papers in library cite

Nice paper that introduced an important dataset. Not much else though.

[10]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[11]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[12]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[13]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[14]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[15]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[16]Bidirectional Attention Flow for Machine Comprehension

M. Seo, A. Kembhavi, Ali Farhadi, Hananneh Hajishirzi - 2017

13 papers in library cite

It's alright but the method seems absurdly complex. Maybe I am a bit biased because it's like the 20th paper that I read with attention + LSTMs...

[17]Automatically Constructing a Corpus of Sentential Paraphrases

W. Dolan, Chris Brockett - 2005

9 papers in library cite

Small dataset, questionable methodology, not useful for training models

[18]The Winograd Schema Challenge

Hector J. Levesque, E. Davis, Leora Morgenstern - 2011

13 papers in library cite

It's amazing to see a paper that is very easy to read, opinionated, but also introducing an important contribution to the field. I love that they introduced this challenge as an alternative to the Imitation Game.

[19]Annotation Artifacts in Natural Language Inference Data

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Richard Schwartz, S. Bowman, Noah A. Smith - 2018

6 papers in library cite

I love papers that show flaws in current methodology/datasets. Very nice for pointing it out!

[20]One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

C. Chelba, Tomas Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, Tony Robinson - 2013

13 papers in library cite

It's somewhat shallow, but I can see the importance of this paper.

[21]Learned in Translation: Contextualized Word Vectors

B. Mccann, J. Bradbury, Caiming Xiong, Richard Socher - 2017

14 papers in library cite

Doesn't seem too revolutionary tbh. It's a nice methodology though, and it's nice that they do it in a supervised way

[22]Reasoning About Entailment With Neural Attention

Tim Rocktaschel, Edward Grefenstette, K. Hermann, T. Kocisky, Phil Blunsom - 2016

5 papers in library cite

It's nice that they are SotA on top of SNLI, but they just apply existing methodologies.

[23]The Natural Language Decathlon: Multitask Learning as Question Answering

Richard Socher - 2018

9 papers in library cite

Very nice results, but TBH the architecture is waaaay to complicated

[24]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[25]Neural Network Acceptability Judgments

Alex Warstadt, A. Singh, S. Bowman - 2018

8 papers in library cite

CoLA dataset

[26]Allennlp: A Deep Semantic Natural Language Processing Platform

Matt Gardner, J. Grus, M. Neumann, Oyvind Tafjord, P. Dasigi, N. Liu, M. Peters, M. Schmitz, Luke Zettlemoyer - 2018

5 papers in library cite

Framework from alenai

[27]Senteval: An Evaluation Toolkit for Universal Sentence Representations

Alexis Conneau, Douwe Kiela - 2018

5 papers in library cite

Sentence embedding eval toolkit?

[28]Hypothesis Only Baselines in Natural Language Inference

A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, B. V. Durme - 2018

5 papers in library cite

Using only the hypothesis can solve the NLI task (it shouldn'!)

[29]Learning General Purpose Distributed Sentence Representations via Large Scale Multi-Task Learning

S. Subramanian, A. Trischler, Yoshua Bengio, C. Pal - 2018

4 papers in library cite

Bengio

[30]The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task

Richard Schwartz, Maarten Sap, I. Konstas, L. Zilles, Yejin Choi, Noah A. Smith - 2017

3 papers in library cite

People write differently based on the prompt and that makes it easy to distinguish things

[31]Seeing Stars: Exploiting Class Relationships for Sentiment Categorization With Respect to Rating Scales

Bo Pang, L. Lee - 2005

13 papers in library cite

[32]A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts

Bo Pang, L. A. Lee, L. Lillian - 2004

8 papers in library cite

[33]Annotating Expressions of Opinions and Emotions in Language

J. Wiebe, T. Wilson, T. Theresa, C. A. Cardie, C. Claire - 2005

7 papers in library cite

[34]The Fifth PASCAL Recognizing Textual Entailment Challenge

L. Bentivogli, Peter Clark, Ido Dagan, D. Giampiccolo - 2009

7 papers in library cite

[35]The Third PASCAL Recognizing Textual Entailment Challenge

D. Giampiccolo, Bernardo Magnini, Ido Dagan, B. Dolan - 2007

7 papers in library cite

[36]Mining and Summarizing Customer Reviews

M. Hu, B. A. Liu, B. Bing - 2004

6 papers in library cite

[37]SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation

D. Cer, M. Diab, E. Agirre, I. L. Gazpio, L. Specia - 2017

6 papers in library cite

[38]The Second PASCAL Recognising Textual Entailment Challenge

R. B. Haim, Ido Dagan, B. Dolan, L. Ferro, D. Giampiccolo, Bernardo Magnini, I. Szpektor - 2006

6 papers in library cite

[39]A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks

K. Hashimoto, Caiming Xiong, Y. Tsuruoka, Richard Socher - 2016

5 papers in library cite

[40]The trec-8 Question Answering Track Evaluation

E. M. Voorhees, D. M. Tice - 1999

5 papers in library cite

[41]Bag of Tricks for Efficient Text Classification

Armand Joulin, E. Grave, Piotr Bojanowski, Tomas Mikolov - 2017

4 papers in library cite

[42]DisSent: Learning Sentence Representations From Explicit Discourse Relations

Allen Nie, E. Bennett, N. Goodman - 2017

4 papers in library cite

[43]Deep Multi-Task Learning With Low Level Tasks Supervised at Lower Layers

A. Sogaard, Y. Goldberg - 2016

3 papers in library cite

[44]Non-Entailed Subsequences as a Challenge for Natural Language Inference

R. T. Mccoy, Tal Linzen - 2019

3 papers in library cite

[45]Using the Framework

R. Cooper, D. Crouch, J. Eijck, C. Fox, J. Genabith, J. Jaspars, H. Kamp, D. Milward, M. Pinkal, M. Poesio, S. Pulman, T. Briscoe, H. Maier, K. Konrad - 1996

3 papers in library cite

[46]Inference Is Everything: Recasting Semantic Resources Into a Unified Evaluation Framework

A. White, P. Rastogi, K. Duh, B. Durme - 2017

2 papers in library cite

[47]Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment

M. Tsuchiya - 2018

2 papers in library cite

[48]Sluice Networks: Learning What to Share Between Loosely Related Tasks

Sebastian Ruder, J. Bingel, I. Augenstein, A. Sogaard - 2017

2 papers in library cite

[49]Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task

A. Ettinger, S. Rao, H. D. Iii, E. Bender - 2017

2 papers in library cite

[50]Comparing Two K-Category Assignments by a K-Category Correlation Coefficient

J. Gorodkin - 2004

1 paper in library cites

[51]Comparison of the Predicted and Observed Secondary Structure of t4 Phage Lysozyme

B. Matthews - 1975

1 paper in library cites

[52]Transforming Question Answering Datasets Into Natural Language Inference Datasets

D. Demszky, K. Guu, Percy Liang - 2018

1 paper in library cites

Cited by

26

papers in your library

Cites

30

papers in your library

Read

on August 8, 2025

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

Tags

Paper Aliases

No aliases