Papperoni

2018

A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman

Open PDF Google Scholar

citations

Cite Score

75

AI summary

This paper introduces the MultiNLI corpus, a 433k-example dataset for natural language inference, covering ten genres of written and spoken English; evaluation using existing models shows that this corpus represents a more difficult task than the Stanford NLI corpus.

Main Contributions

Introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus for training sentence understanding models
The corpus contains 433k examples from ten distinct genres of written and spoken English
The corpus enables evaluating cross-genre domain adaptation
The corpus is more difficult than the Stanford NLI corpus

Abstract

This paper introduces the Multi-Genre Natural Language Inference (MultiNLI) corpus, a dataset designed for use in the development and evaluation of machine learning models for sentence understanding. At 433k examples, this resource is one of the largest corpora available for natural language inference (a.k.a. recognizing textual entailment), improving upon available resources in both its coverage and difficulty. MultiNLI accomplishes this by offering data from ten distinct genres of written and spoken English, making it possible to evaluate systems on nearly the full complexity of the language, while supplying an explicit setting for evaluating cross-genre domain adaptation. In addition, an evaluation using existing machine learning models designed for the Stanford NLI corpus shows that it represents a substantially more difficult task than does that corpus, despite the two showing similar levels of inter-annotator agreement.

Citation Graph

Loading graph...

References [36]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[3]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[4]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[5]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[6]Visualizing and Understanding Convolutional Networks

Matthew D. Zeiler, Rob Fergus - 2014

15 papers in library cite

Very good explanation and visualization of CNNs, and also nice that they use their findings to improve the performance. The ablation study is also nice.

[7]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[8]DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

J. Donahue, Y. Jia, Oriol Vinyals, J. Hoffman, N. Zhang, E. Tzeng, Trevor Darrell - 2014

15 papers in library cite

Very nice paper. First I've seen (and based on the text, first ever) about feature extraction for images. It's very nice to see embeddings doing SotA

[9]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[10]The PASCAL Recognising Textual Entailment Challenge

Ido Dagan, O. Glickman, Bernardo Magnini - 2005

19 papers in library cite

It's very nice how they had the foresight to create a challenge that became relevant like 10 years later.

[11]Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes - 2017

11 papers in library cite

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

[12]A Decomposable Attention Model for Natural Language Inference

A. P. Parikh, O. Tackstrom, Dipanjan Das, Jakob Uszkoreit - 2016

11 papers in library cite

Very nice alternative to the common LSTM encoder-decoder architecture! Seems similar o the Transformers arch in the sense that they don't use RNNs. Nice that they analyze computational complexity as well.

[13]A Sick Cure for the Evaluation of Compositional Distributional Semantic Models

Marco Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, R. Z. Elli - 2014

7 papers in library cite

Just a basic dataset paper. Nothing much.

[14]Enhanced LSTM for Natural Language Inference

Qinlang Chen, X. Zhu, Z. H. Ling, S. Wei, H. Jiang, D. Inkpen - 2017

5 papers in library cite

SotA NLI

[15]Learning Natural Language Inference With LSTM

Shijie Wang, J. J. Jiang - 2016

3 papers in library cite

NLI with LSTM - used as a baseline in the decomposable attention paper

[16]How Transferable Are Neural Networks in NLP Applications?

L. Mou, Z. Meng, R. Yan, G. Li, Yiheng Xu, Li Zhang, Z. Jin - 2016

3 papers in library cite

Seems like this said that pre-training didnt work

[17]Accurate Unlexicalized Parsing

Dan Klein, Christopher D. Manning - 2003

7 papers in library cite

[18]Semeval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences Through Semantic Relatedness and Textual Entailment

Marco Marelli, L. Bentivogli, M. Baroni, R. Bernardi, S. Menini, R. Zamparelli - 2014

7 papers in library cite

[19]A Fast Unified Model for Parsing and Sentence Understanding

S. Bowman, J. Gauthier, Abhinav Rastogi, R. Gupta, C. Manning, Christopher Potts - 2016

5 papers in library cite

[20]Entailment, Intensionality and Text Understanding

C. Condoravdi, D. Crouch, V. D. Paiva, R. Stolle, D. G. Bobrow - 2003

5 papers in library cite

[21]From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference Over Event Descriptions

P. Young, A. L. M. Hodosh, J. Hockenmaier - 2014

5 papers in library cite

[22]A Natural Logic Inference System

Y. Fyodorov, Y. Winter, N. Francez - 2000

4 papers in library cite

[23]An Extended Model of Natural Logic

B. Maccartney, Christopher D. Manning - 2009

4 papers in library cite

[24]Neural Semantic Encoders

H. Yu, T. Munkhdalai - 2017

4 papers in library cite

[25]Recognising Textual Entailment With Logical Inference

J. Bos, Katja Markert - 2005

4 papers in library cite

[26]Finding Contradictions in Text

M. C. D. Marneffe, A. N. Rafferty, Christopher D. Manning - 2008

3 papers in library cite

[27]Natural Language Inference by Tree-Based Convolution and Heuristic Matching

L. Mou, M. Rui, G. Li, Yiheng Xu, Li Zhang, R. Yan, Z. Jin - 2016

3 papers in library cite

[28]An American National Corpus: A Proposal

C. Fillmore, N. Ide, Dan Jurafsky, C. Macleod - 1998

1 paper in library cites

[29]Integrating Linguistic Resources: The national corpus Model

N. Ide, K. Suderman - 2006

1 paper in library cites

[30]Quantification

A. Szabolcsi - 2010

1 paper in library cites

[31]Semantics and Philosophy, New York, NY: New York University Press, Chapter Pragmatic Presupposition

R. Stalnaker - 1974

1 paper in library cites

[32]Semantics in Generative Grammar

I. Heim, A. Kratzer - 1998

1 paper in library cites

[33]The American National Corpus: A Standardized Resource for American English

C. Macleod, N. Ide, R. Grishman - 2000

1 paper in library cites

[34]The American National Corpus: A Standardized Resource for American English

C. Macleod, N. Ide, R. Grishman - 2000

1 paper in library cites

[35]The Cambridge Handbook of Formal Semantics, Cambridge University Press, Chapter the Semantics/Pragmatics Interface

P. Schlenker - 2016

1 paper in library cites

[36]The Repeval 2017 Shared Task: Multi-Genre Natural Language Inference With Sentence Representations

Nikita Nangia, A. Williams, A. Lazaridou, Samuel R. Bowman - 2017

1 paper in library cites

Cited by

19

papers in your library

Cites

16

papers in your library

Read

on October 25, 2025

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

Tags

Paper Aliases

No aliases