Papperoni

2017

Supervised Learning of Universal Sentence Representations From Natural Language Inference Data

Alexis Conneau, Douwe Kiela, Holger Schwenk, L. Barrault, Antoine Bordes

Open PDF Google Scholar

citations

Cite Score

62

AI summary

This paper introduces a new method for learning universal sentence representations by training a BiLSTM-based encoder on the SNLI dataset, achieving state-of-the-art results on a variety of transfer tasks compared to unsupervised methods like SkipThought.

Main Contributions

Demonstrates that supervised learning on the SNLI dataset can produce high-quality universal sentence representations.
Shows that a BiLSTM architecture with max pooling, trained on SNLI, outperforms unsupervised methods like SkipThought on various transfer tasks.
Introduces a sentence evaluation toolkit called SentEval for automating evaluation on transfer tasks.
Achieves state-of-the-art results on several transfer tasks using sentence embeddings trained on SNLI.
Shows that using a larger coverage dataset (MultiNLI) for training sentence encoders helps learn even better general representations.

Abstract

Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features. Efforts to obtain embeddings for larger chunks of text, such as sentences, have however not been so successful. Several attempts at learning unsupervised representations of sentences have not reached satisfactory enough performance to be widely adopted. In this paper, we show how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks. Much like how computer vision uses ImageNet to obtain features, which can then be transferred to other tasks, our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks. Our encoder is publicly available.

Citation Graph

Loading graph...

References [41]

Sort:

Filter:

[1]Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2016

20 papers in library cite

This is simply amazing. Very very simple idea, totally revolutionary. No maths, just "it works!". Amazing.

[2]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[3]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[4]ImageNet: A Large-Scale Hierarchical Image Database

J. Deng, W. Dong, Richard Socher, L. J. Li, K. Li, Li Fei Fei - 2009

28 papers in library cite

Very nice idea and huge impact!

[5]Microsoft COCO: Common Objects in Context

T. Y. Lin, M. Maire, S. Belongie, James Hays, Pietro Perona, D. Ramanan, Piotr Dollar, C. L. Zitnick - 2014

14 papers in library cite

I liked this paper a lot. It's a bit long and I was already a bit tired, but it was nice overall.

[6]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[7]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[8]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[9]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[10]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[11]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[12]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[13]On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Kyunghyun Cho, B. V. Merrienboer, D. Bahdanau, Yoshua Bengio - 2014

9 papers in library cite

This just builds upon other work, comparing and analyzing previous architectures. This faces a specific problem of the time: long sentences. Not too relevant today.

[14]Deepface: Closing the Gap to Human-Level Performance in Face Verification

Y. Taigman, Michael Yang, Marc'aurelio Ranzato, Lior Wolf - 2014

5 papers in library cite

Very impressive results but boring overall. I think the main thing is that they were the first to use CNN, but it seems like the most important part of their method was the 3D alignment (which is nice, but out of scope for me)

[15]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[16]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[17]A Broad-Coverage Challenge Corpus for Sentence Understanding Through Inference

A. Williams, Nikita Nangia, S. Bowman - 2018

19 papers in library cite

Very nice paper and cool dataset - good thing they expanded SNLI. Also, they at least tried to have a good baseline, and comparisons of domains are nice.

[18]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[19]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[20]A Structured Self-Attentive Sentence Embedding

Zongyu Lin, M. Feng, C. D. Santos, M. Yu, Bing Xiang, B. Zhou, Yoshua Bengio - 2017

2 papers in library cite

It's a bit poorly written and I don't like their 2D embeddings, but the visualizations are nice :)

[21]A Sick Cure for the Evaluation of Compositional Distributional Semantic Models

Marco Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, R. Z. Elli - 2014

7 papers in library cite

Just a basic dataset paper. Nothing much.

[22]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[23]Deep Visual-Semantic Alignments for Generating Image Descriptions

A. Karpathy, Li Fei Fei - 2014

6 papers in library cite

Early multimodality

[24]Vqa: Visual Question Answering

S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, D. Parikh - 2015

6 papers in library cite

New task?

[25]CNN Features Off-the-Shelf: An Astounding Baseline for Recognition

A. Razavian, H. Azizpour, J. Sullivan, S. Carlsson - 2014

6 papers in library cite

CNN feature extraction

[26]Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

K. S. Tai, Richard Socher, Christopher D. Manning - 2015

6 papers in library cite

Baseline for skip-thoughts paper

[27]A Simple but Tough-to-Beat Baseline for Sentence Embeddings

S. Arora, Yiqing Liang, T. Ma - 2017

4 papers in library cite

[28]Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics

M. Hodosh, P. Young, J. Hockenmaier - 2013

4 papers in library cite

Lays some of the groundwork for the captioning stuff

[29]Senteval: An Evaluation Toolkit for Universal Sentence Representations

Alexis Conneau, Douwe Kiela - 2018

5 papers in library cite

Sentence embedding eval toolkit?

[30]Towards Universal Paraphrastic Sentence Embeddings

J. Wieting, Mohit Bansal, Kevin Gimpel, K. A. Livescu, K. Karen - 2015

4 papers in library cite

[31]Order-Embeddings of Images and Language

I. Vendrov, R. Kiros, Sanja Fidler, R. Urtasun - 2016

4 papers in library cite

learning a joint embedding space of images and sentences

[32]How Transferable Are Neural Networks in NLP Applications?

L. Mou, Z. Meng, R. Yan, G. Li, Yiheng Xu, Li Zhang, Z. Jin - 2016

3 papers in library cite

Seems like this said that pre-training didnt work

[33]Illinois-lh: A Denotational and Distributional Approach to Semantics

A. Lai, J. Hockenmaier - 2014

5 papers in library cite

[34]SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

E. Agirre, C. Banea, C. Cardie, D. M. Cer, M. T. Diab, A. G. Agirre, W. Guo, R. Mihalcea, G. Rigau, J. Wiebe - 2014

4 papers in library cite

[35]Discriminative Improvements to Distributional Sentence Similarity

Yangfeng Ji, J. Eisenstein - 2013

3 papers in library cite

[36]Self-Adaptive Hierarchical Sentence Model

H. Zhao, Z. L. Lu, Z. A. Poupart, P. Pascal - 2015

3 papers in library cite

[37]Learning Natural Language Inference Using Bidirectional LSTM Model and Inner-Attention

Yibo Liu, C. Sun, L. Lin, Xinpeng Wang - 2016

2 papers in library cite

[38]Learning Visually Grounded Sentence Representations

Douwe Kiela, Alexis Conneau, A. Jabri, M. Nickel - 2017

2 papers in library cite

[39]Multimodal Convolutional Neural Networks for Matching Image and Sentence

L. Ma, Z. L. Lu, L. Shang, H. Li - 2015

1 paper in library cites

[40]PPDB: The Paraphrase Database

J. Ganitkevitch, B. V. Durme, Chris Callison Burch - 2013

1 paper in library cites

[41]The Multiverse Loss for Robust Transfer Learning

E. Littwin, Lior Wolf - 2016

1 paper in library cites

Cited by

11

papers in your library

Cites

32

papers in your library

Read

on October 24, 2025

It's nice. Maybe the first to do NLI right (after the SNLI paper tried but failed miserably). It is simple and effective. After that people started "performance maxxxing"

Tags

Paper Aliases

No aliases