Papperoni

2016

A Decomposable Attention Model for Natural Language Inference

A. P. Parikh, O. Tackstrom, Dipanjan Das, Jakob Uszkoreit

Open PDF Google Scholar

citations

Cite Score

52

AI summary

This paper introduces a decomposable attention model for natural language inference, achieving state-of-the-art results on the SNLI dataset with fewer parameters than previous LSTM-based approaches, by aligning local text substructure and aggregating the information using attention mechanisms.

Main Contributions

Introduces a decomposable attention model for NLI that decomposes the problem into subproblems solved separately.
Achieves state-of-the-art results on the SNLI dataset using fewer parameters compared to LSTM-based models.
Leverages attention mechanisms to align local text substructure and aggregate information effectively.
Demonstrates the effectiveness of pairwise comparisons over global sentence-level representations for NLI.
Shows that the approach is trivially parallelizable which can be useful at test time.

Abstract

We propose a simple neural architecture for natural language inference. Our approach uses attention to decompose the problem into subproblems that can be solved separately, thus making it trivially parallelizable. On the Stanford Natural Language Inference (SNLI) dataset, we obtain state-of-the-art results with almost an order of magnitude fewer parameters than previous work and without relying on any word-order information. Adding intra-sentence attention that takes a minimum amount of order into account yields further improvements.

Citation Graph

Loading graph...

References [31]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[3]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[4]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[5]Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

John Duchi, Elad Hazan, Yoram Singer - 2011

19 papers in library cite

I actually skimmed through most of this. It's not a bad paper, but it's a math paper, not AI.

[6]TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

M. Abadi, Akshat Agarwal, P. Barham, E. Brevdo, Ziru Chen, C. Citro, G. Corrado, A. Davis, Jeffrey Dean, M. Devin, Sanjay Ghemawat, I. Goodfellow, A. Harp, Geoffrey Irving, M. Isard, Y. Jia, R. Jozefowicz, Lukasz Kaiser, M. Kudlur, J. Levenberg, D. Mane, R. Monga, S. Moore, D. Murray, Christopher Olah, M. Schuster, J. Shlens, B. Steiner, Ilya Sutskever, K. Talwar, P. Tucker, Vincent Vanhoucke, V. Vasudevan, F. Viegas, Oriol Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, Xiaoqiang Zheng - 2015

11 papers in library cite

This should be the golden standard to what framework papers should be. It's large, but it's not boring at all. Explains the core concepts while not going too deep as to describe unimportant things; explains design decisions and shortcomings... overall amazing

[7]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[8]Handwritten Digit Recognition With a Back-Propagation Network

Yann Lecun, B. Boser, John S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel - 1990

10 papers in library cite

Just a minor derivation of the previous paper, adding more layers to the net. Pretty much the same exact text than the other one. Uninteresting if you read the other.

[9]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[10]Long Short-Term Memory-Networks for Machine Reading

Mirella Lapata - 2016

8 papers in library cite

I read this more as an example of intra-attention, but this is not the main focus of the paper. I think visualization/explanation is a bit bad, and it doesn't seem too impactful. I kept thinking that this is starting to get too complicated, and indeed it was surpassed by transformers right after that.

[11]Reasoning About Entailment With Neural Attention

Tim Rocktaschel, Edward Grefenstette, K. Hermann, T. Kocisky, Phil Blunsom - 2016

5 papers in library cite

It's nice that they are SotA on top of SNLI, but they just apply existing methodologies.

[12]Convolutional Neural Network Architectures for Matching Natural Language Sentences

B. Hu, Z. L. Lu, H. Li, Qinlang Chen - 2014

2 papers in library cite

Previous work on NLI

[13]ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs

W. Yin, Hinrich Schutze, Bing Xiang, B. Zhou - 2016

1 paper in library cites

NLI with CNN?

[14]Order-Embeddings of Images and Language

I. Vendrov, R. Kiros, Sanja Fidler, R. Urtasun - 2016

4 papers in library cite

learning a joint embedding space of images and sentences

[15]Learning Natural Language Inference With LSTM

Shijie Wang, J. J. Jiang - 2016

3 papers in library cite

NLI with LSTM - used as a baseline in the decomposable attention paper

[16]A Fast Unified Model for Parsing and Sentence Understanding

S. Bowman, J. Gauthier, Abhinav Rastogi, R. Gupta, C. Manning, Christopher Potts - 2016

5 papers in library cite

[17]Statistical Machine Translation

P. Koehn - 2010

5 papers in library cite

[18]An Extended Model of Natural Logic

B. Maccartney, Christopher D. Manning - 2009

4 papers in library cite

[19]Recognising Textual Entailment With Logical Inference

J. Bos, Katja Markert - 2005

4 papers in library cite

[20]Natural Language Inference by Tree-Based Convolution and Heuristic Matching

L. Mou, M. Rui, G. Li, Yiheng Xu, Li Zhang, R. Yan, Z. Jin - 2016

3 papers in library cite

[21]Paraphrase Identification as Probabilistic Quasi-Synchronous Recognition

Dipanjan Das, Noah A. Smith - 2009

3 papers in library cite

[22]Paraphrase-Driven Learning for Open Question Answering

A. Fader, Luke Zettlemoyer, Oren Etzioni - 2013

3 papers in library cite

[23]A Brief History of Natural Logic

J. V. Benthem - 2008

2 papers in library cite

[24]Classification of Semantic Relations by Humans and Machines

E. Marsi, E. Krahmer - 2005

2 papers in library cite

[25]Semantic Theory

J. J. Katz - 1972

2 papers in library cite

[26]A Discourse Commitment-Based Framework for Recognizing Textual Entailment

A. Hickl, J. Bensley - 2007

1 paper in library cites

[27]A Phrase-Based Alignment Model for Natural Language Inference

B. Maccartney, M. Galley, Christopher D. Manning - 2008

1 paper in library cites

[28]Discriminative Learning Over Constrained Latent Representations

M. Chang, D. Goldwasser, Dan Roth, Vivek Srikumar - 2010

1 paper in library cites

[29]Learning to Recognize Features of Valid Textual Entailments

B. Maccartney, T. Grenager, M. D. Marneffe, D. Cer, Christopher D. Manning - 2006

1 paper in library cites

[30]Robust Textual Inference via Graph Matching

A. D. Haghighi, Andrew Y. Ng, Christopher D. Manning - 2005

1 paper in library cites

[31]Semantic Parsing as Machine Translation

Jacob Andreas, A. Vlachos, S. Clark - 2013

1 paper in library cites

Cited by

11

papers in your library

Cites

15

papers in your library

Read

on August 4, 2025

Very nice alternative to the common LSTM encoder-decoder architecture! Seems similar o the Transformers arch in the sense that they don't use RNNs. Nice that they analyze computational complexity as well.

Tags

Paper Aliases

No aliases