Papperoni

2017

A Structured Self-Attentive Sentence Embedding

Zongyu Lin, M. Feng, C. D. Santos, M. Yu, Bing Xiang, B. Zhou, Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

63

AI summary

This paper introduces a self-attention mechanism to create a fixed-size matrix sentence embedding, enabling interpretability and improved performance on author profiling, sentiment classification, and textual entailment tasks, achieving significant gains compared to other sentence embedding methods.

Main Contributions

Introduces a self-attention mechanism for fixed-size matrix sentence embeddings.
Presents a 2-D matrix embedding where each row attends to a different part of the sentence.
Proposes a special regularization term to encourage diversity in attention.
Provides a visualization method to understand what the embedding encodes.
Achieves significant performance gains on author profiling, sentiment classification, and textual entailment tasks.

Abstract

This paper proposes a new model for extracting an interpretable sentence embedding by introducing self-attention. Instead of using a vector, we use a 2-D matrix to represent the embedding, with each row of the matrix attending on a different part of the sentence. We also propose a self-attention mechanism and a special regularization term for the model. As a side effect, the embedding comes with an easy way of visualizing what specific parts of the sentence are encoded into the embedding. We evaluate our model on 3 different tasks: author profiling, sentiment classification and textual entailment. Results show that our model yields a significant performance gain compared to other sentence embedding methods in all of the 3 tasks.

Citation Graph

Loading graph...

References [36]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[3]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[4]Convolutional Neural Networks for Sentence Classification

Yoon Kim - 2014

8 papers in library cite

It's nice, goes straight to the point. I can see why it has tons of citations. However, I am not sure it was as impactful as 20k citations.

[5]Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2014

11 papers in library cite

It's a good paper but results are veeeery underwhelming.

[6]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[7]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[8]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[9]A Large Annotated Corpus for Learning Natural Language Inference

Samuel R. Bowman, G. Angeli, Christopher Potts, Christopher D. Manning - 2015

25 papers in library cite

Dataset collection is ok. The model that they create seems very low effort.

[10]A Convolutional Neural Network for Modelling Sentences

Phil Blunsom, Edward Grefenstette, N. Kalchbrenner - 2014

7 papers in library cite

I am usually biased against CNNs, but this was very good! Very well motivated and simple/intuitive explanations!

[11]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[12]A Decomposable Attention Model for Natural Language Inference

A. P. Parikh, O. Tackstrom, Dipanjan Das, Jakob Uszkoreit - 2016

11 papers in library cite

Very nice alternative to the common LSTM encoder-decoder architecture! Seems similar o the Transformers arch in the sense that they don't use RNNs. Nice that they analyze computational complexity as well.

[13]Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning - 2011

10 papers in library cite

Very nice and very simple method. I like this waaay better than the other Stanford recursive autoencoder papers.

[14]Long Short-Term Memory-Networks for Machine Reading

Mirella Lapata - 2016

8 papers in library cite

I read this more as an example of intra-attention, but this is not the main focus of the paper. I think visualization/explanation is a bit bad, and it doesn't seem too impactful. I kept thinking that this is starting to get too complicated, and indeed it was surpassed by transformers right after that.

[15]Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen - 2016

12 papers in library cite

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

[16]Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

K. S. Tai, Richard Socher, Christopher D. Manning - 2015

6 papers in library cite

Baseline for skip-thoughts paper

[17]Order-Embeddings of Images and Language

I. Vendrov, R. Kiros, Sanja Fidler, R. Urtasun - 2016

4 papers in library cite

learning a joint embedding space of images and sentences

[18]A Fast Unified Model for Parsing and Sentence Understanding

S. Bowman, J. Gauthier, Abhinav Rastogi, R. Gupta, C. Manning, Christopher Potts - 2016

5 papers in library cite

[19]Neural Semantic Encoders

H. Yu, T. Munkhdalai - 2017

4 papers in library cite

[20]Neural Tree Indexers for Text Understanding

H. Yu, T. Munkhdalai - 2017

4 papers in library cite

[21]Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering

P. L. Li, Wentao Li, Z. He, Xinpeng Wang, Yue Cao, Jingren Zhou, Weixin Xu - 2016

3 papers in library cite

[22]Discriminative Neural Sentence Modeling by Tree-Based Convolution

L. Mou, H. Peng, G. Li, Yiheng Xu, Li Zhang, Z. Jin - 2015

3 papers in library cite

[23]Natural Language Inference by Tree-Based Convolution and Heuristic Matching

L. Mou, M. Rui, G. Li, Yiheng Xu, Li Zhang, R. Yan, Z. Jin - 2016

3 papers in library cite

[24]Attentive Pooling Networks

C. D. Santos, M. Tan, Bing Xiang, B. Zhou - 2016

2 papers in library cite

[25]Convolutional Neural Network for Paraphrase Identification

W. Yin, Hinrich Schutze - 2015

2 papers in library cite

[26]Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts

C. N. D. Santos, M. Gatti - 2014

2 papers in library cite

[27]Learning Natural Language Inference Using Bidirectional LSTM Model and Inner-Attention

Yibo Liu, C. Sun, L. Lin, Xinpeng Wang - 2016

2 papers in library cite

[28]Not All Contexts Are Created Equal: Better Word Representations With Variable Attention

W. Ling, L. C. Cheng, Y. Tsvetkov, S. Amir - 2015

2 papers in library cite

[29]Theano: A Python Framework for Fast Computation of Mathematical Expressions

T. T. D. Team, R. A. Rfou, G. Alain, Amjad Almahairi, C. Angermueller, D. Bahdanau, Nicolas Ballas, F. Bastien, J. Bayer, A. Belikov - 2016

2 papers in library cite

[30]A Batch-Normalized Recurrent Network for Sentiment Classification

H. Margarit, R. Subramaniam - 2016

1 paper in library cites

[31]Applying Deep Learning to Answer Selection: A Study and an Open Task

M. Feng, Bing Xiang, M. R. Glass, Lisa Wang, B. Zhou - 2015

1 paper in library cites

[32]Deep Sentence Embedding Using Long Short-Term Memory Networks: Analysis and Application to Information Retrieval

H. Palangi, L. Deng, Y. Shen, Jianfeng Gao, X. He, Jixuan Chen, X. Song, R. Ward - 2016

1 paper in library cites

[33]Dependency-Based Convolutional Neural Networks for Sentence Embedding

M. Ma, L. Huang, Bing Xiang, B. Zhou - 2015

1 paper in library cites

[34]Improved Representation Learning for Question Answer Matching

M. Tan, C. D. Santos, Bing Xiang, B. Zhou - 2016

1 paper in library cites

[35]Learning to Relate Images

R. Memisevic - 2013

1 paper in library cites

[36]Sequential Short-Text Classification With Recurrent and Convolutional Neural Networks

J. Y. Lee, F. Dernoncourt - 2016

1 paper in library cites

Cited by

2

papers in your library

Cites

17

papers in your library

Read

on August 6, 2025

It's a bit poorly written and I don't like their 2D embeddings, but the visualizations are nice :)

Tags

Paper Aliases

No aliases