Papperoni

2018

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Yejin Choi

Open PDF Google Scholar

citations

Cite Score

35

AI summary

This paper introduces SWAG, a new dataset with 113k multiple choice questions for grounded commonsense inference, unifying natural language inference and commonsense reasoning. To address the recurring challenges of the annotation artifacts and human biases found in many existing datasets, they propose Adversarial Filtering (AF).

Main Contributions

Introduces the task of grounded commonsense inference, unifying natural language inference and commonsense reasoning.
Presents SWAG, a new dataset with 113k multiple choice questions.
Proposes Adversarial Filtering (AF), a novel procedure that constructs a de-biased dataset by iteratively training an ensemble of stylistic classifiers, and using them to filter the data.
Demonstrates that while humans can solve the resulting inference problems with high accuracy (88%), various competitive models struggle on our task.
Provides comprehensive analysis that indicates significant opportunities for future research.

Abstract

Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine"). In this paper, we introduce the task of grounded commonsense inference, unifying natural language inference and commonsense reasoning. We present SWAG, a new dataset with 113k multiple choice questions about a rich spectrum of grounded situations. To address the recurring challenges of the annotation artifacts and human biases found in many existing datasets, we propose Adversarial Filtering (AF), a novel procedure that constructs a de-biased dataset by iteratively training an ensemble of stylistic classifiers, and using them to filter the data. To account for the aggressive adversarial filtering, we use state-of-the-art language models to massively oversample a diverse set of potential counterfactuals. Empirical results demonstrate that while humans can solve the resulting inference problems with high accuracy (88%), various competitive models struggle on our task. We provide comprehensive analysis that indicates significant opportunities for future research.

Citation Graph

Loading graph...

References [60]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[4]Deep Contextualized Word Representations

M. E. Peters, M. Neumann, M. Iyyer, Matt Gardner, C. Clark, K. Lee, L. S. Zettlemoyer - 2018

27 papers in library cite

I didn't really like the approach. Seems a bit derivative TBH. BERT seems more elegant.

[5]GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

A. Wang, A. Singh, J. Michael, F. Hill, Omer Levy, Samuel R. Bowman - 2018

26 papers in library cite

I like it, but it's just a mesh of different existing datasets and F1 score. Nothing new really but I get why it's important

[6]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[7]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[8]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[9]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[10]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[11]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[12]A Decomposable Attention Model for Natural Language Inference

A. P. Parikh, O. Tackstrom, Dipanjan Das, Jakob Uszkoreit - 2016

11 papers in library cite

Very nice alternative to the common LSTM encoder-decoder architecture! Seems similar o the Transformers arch in the sense that they don't use RNNs. Nice that they analyze computational complexity as well.

[13]A Sick Cure for the Evaluation of Compositional Distributional Semantic Models

Marco Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, R. Z. Elli - 2014

7 papers in library cite

Just a basic dataset paper. Nothing much.

[14]Annotation Artifacts in Natural Language Inference Data

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Richard Schwartz, S. Bowman, Noah A. Smith - 2018

6 papers in library cite

I love papers that show flaws in current methodology/datasets. Very nice for pointing it out!

[15]The LAMBADA dataset: Word Prediction Requiring a Broad Discourse Context

D. Paperno, German Kruszewski, A. Lazaridou, N. Q. Pham, R. Bernardi, S. Pezzelle, M. Baroni, G. Boleda, Raquel Fernandez - 2016

12 papers in library cite

Very nice paper - very interesting methodology to building it and very good when they bring a dataset that is meant to make machines to fail

[16]Using the Output Embedding to Improve Language Models

O. Press, Lior Wolf - 2017

7 papers in library cite

I did not like this paper at all - The paper is not bad, it's just that I expected *way* more. Good results but uninteresting

Reference title contains 'et al'

[17]Latent Dirichlet Allocation

D. M. Blei, Andrew Y. Ng, Michael I. Jordan - 2003

10 papers in library cite

30 pages; LDA

[18]ActivityNet: A large-scale video benchmark for human activity understanding

F. C. Heilbron, V. Escorcia, B. Ghanem, J. C. Niebles - 2015

1 paper in library cites

Dataset of videos

[19]Mitigating Unwanted Biases With Adversarial Learning

B. H. Zhang, B. Lemoine, M. Mitchell - 2018

1 paper in library cites

Very nice intro to adversarial learning

[20]Dense-Captioning Events in Videos

R. Krishna, K. Hata, F. Ren, Li Fei Fei, J. C. Niebles - 2017

2 papers in library cite

[21]Enhanced LSTM for Natural Language Inference

Qinlang Chen, X. Zhu, Z. H. Ling, S. Wei, H. Jiang, D. Inkpen - 2017

5 papers in library cite

SotA NLI

[22]Allennlp: A Deep Semantic Natural Language Processing Platform

Matt Gardner, J. Grus, M. Neumann, Oyvind Tafjord, P. Dasigi, N. Liu, M. Peters, M. Schmitz, Luke Zettlemoyer - 2018

5 papers in library cite

Framework from alenai

[23]Hypothesis Only Baselines in Natural Language Inference

A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, B. V. Durme - 2018

5 papers in library cite

Using only the hypothesis can solve the NLI task (it shouldn'!)

[24]Movie Description

A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, Hugo Larochelle, Aaron Courville, B. Schiele - 2017

2 papers in library cite

[25]The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task

Richard Schwartz, Maarten Sap, I. Konstas, L. Zilles, Yejin Choi, Noah A. Smith - 2017

3 papers in library cite

People write differently based on the prompt and that makes it easy to distinguish things

[26]Breaking Nli Systems With Sentences That Require Simple Lexical Inferences

M. Glockner, V. Shwartz, Y. Goldberg - 2018

3 papers in library cite

[27]Social Bias in Elicited Natural Language Inferences

R. Rudinger, C. May, B. V. Durme - 2017

3 papers in library cite

[28]Revisiting Visual Question Answering Baselines

A. Jabri, Armand Joulin, Laurens Van Der Maaten - 2016

2 papers in library cite

[29]The Nature of Statistical Learning Theory

V. Vapnik - 1995

9 papers in library cite

Book, 300+ pages

[30]The microsoft research Sentence Completion Challenge

Geoffrey Zweig, C. J. Burges - 2011

6 papers in library cite

[31]Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

H. Inan, K. Khosravi, Richard Socher - 2017

6 papers in library cite

[32]A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories

N. Mostafazadeh, N. Chambers, X. He, D. Parikh, D. Batra, L. Vanderwende, P. Kohli, J. Allen - 2016

5 papers in library cite

[33]Bag of Tricks for Efficient Text Classification

Armand Joulin, E. Grave, Piotr Bojanowski, Tomas Mikolov - 2017

4 papers in library cite

[34]A Framework for Computational Semantics (fracas)

R. Cooper, D. Crouch, J. Eijckl, C. Fox, J. Genabith, J. Japars, H. Kamp, D. Milward, M. Pinkal, M. Poesio - 1996

2 papers in library cite

[35]Meaning and Grammar: An Introduction to Semantics

G. Chierchia, S. M. Ginet - 2001

2 papers in library cite

[36]Multi-Reward Reinforced Summarization With Saliency and Entailment

R. Pasunuru, Mohit Bansal - 2018

2 papers in library cite

[37]Neural motifs: Scene Graph Parsing With Global Context

Rowan Zellers, M. Yatskar, S. Thomson, Yejin Choi - 2018

2 papers in library cite

[38]Pay Attention to the Ending: Strong Neural Baselines for the roc Story Cloze Task

Zhipeng Cai, L. Tu, Kevin Gimpel - 2017

2 papers in library cite

[39]Reasoning With Neural Tensor Networks for Knowledge Base Completion

Richard Socher, Deli Chen, Christopher D. Manning, A. Ng - 2013

2 papers in library cite

[40]A Minimal Span-Based Neural Constituency Parser

M. Stern, Jacob Andreas, Dan Klein - 2017

1 paper in library cites

[41]Are Distributional Representations Ready for the Real World? Evaluating Word Vectors for Grounded Perceptual Meaning

L. Lucy, J. Gauthier - 2017

1 paper in library cites

[42]Commonsense Knowledge Base Completion

Xiang Lisa Li, A. Taheri, L. Tu, Kevin Gimpel - 2016

1 paper in library cites

[43]Conceptnet 5.5: An Open Multilingual Graph of General Knowledge

R. Speer, J. Chin, C. Havasi - 2017

1 paper in library cites

[44]Connotation Frames of Power and Agency in Modern Films

Maarten Sap, M. C. Prasettio, Ari Holtzman, Hannah Rashkin, Yejin Choi - 2017

1 paper in library cites

[45]Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

B. A. Plummer, Lisa Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, Svetlana Lazebnik - 2017

1 paper in library cites

[46]Gender-Distinguishing Features in Film Dialogue

A. Schofield, L. Mehr - 2016

1 paper in library cites

[47]Men Also Like Shopping: Reducing Gender Bias Amplification Using Corpus-Level Constraints

J. Zhao, Tianle Wang, M. Yatskar, V. Ordonez, K. W. Chang - 2017

1 paper in library cites

[48]Multi-Task Video Captioning With Video and Entailment Generation

R. Pasunuru, Mohit Bansal - 2017

1 paper in library cites

[49]Natural Language Inference From Multiple Premises

A. Lai, Yonatan Bisk, J. Hockenmaier - 2017

1 paper in library cites

[50]Ordinal Common-Sense Inference

S. Zhang, R. Rudinger, K. Duh, B. V. Durme - 2017

1 paper in library cites

[51]Scripts, Plans, and Knowledge

R. C. Schank, R. P. Abelson - 1975

1 paper in library cites

[52]Tackling the Story Ending Biases in the Story Cloze Test

R. Sharma, J. Allen, O. Bakhshandeh, N. Mostafazadeh - 2018

1 paper in library cites

[53]The berkeley Framenet Project

C. F. Baker, C. J. Fillmore, J. B. Lowe - 1998

1 paper in library cites

[54]The Ecological Approach to Visual Perception

J. Gibson - 1979

1 paper in library cites

[55]Types of Common-Sense Knowledge Needed for Recognizing Textual Entailment

P. Lobue, A. Yates - 2011

1 paper in library cites

[56]Unsupervised Learning of Narrative Schemas and Their Participants

N. Chambers, Dan Jurafsky - 2009

1 paper in library cites

[57]Visual Madlibs: Fill in the Blank Image Generation and Question Answering

Longhui Yu, E. Park, A. C. Berg, T. L. Berg - 2015

1 paper in library cites

[58]What Will Happen Next? Forecasting Player Moves in Sports Videos

P. Felsen, P. Agrawal, Jitendra Malik - 2017

1 paper in library cites

[59]Who Let the Dogs Out? Modeling Dog Behavior From Visual Data

K. Ehsani, H. Bagherinezhad, Joseph Redmon, R. Mottaghi, Ali Farhadi - 2018

1 paper in library cites

[60]Zero-Shot Activity Recognition With Verb attribute Induction

Rowan Zellers, Yejin Choi - 2017

1 paper in library cites

Cited by

5

papers in your library

Cites

28

papers in your library

Read

on October 25, 2025

Very nice methodology with the adversarial stuff! I think not too relevant though.

Tags

Paper Aliases

No aliases