Papperoni

2015

Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler

Open PDF Google Scholar

citations

Cite Score

65

AI summary

This paper introduces skip-thoughts, a novel unsupervised sentence encoder, trained on the BookCorpus dataset. The encoder-decoder model reconstructs surrounding sentences, mapping sentences with similar properties to similar vector representations. Evaluated on 8 tasks, skip-thoughts demonstrate robust and generic sentence representations.

Main Contributions

Introduces skip-thoughts, an unsupervised sentence encoder that learns generic sentence representations by predicting surrounding sentences.
Presents a vocabulary expansion method to encode words not seen during training, expanding the vocabulary to a million words.
Evaluates skip-thought vectors on 8 tasks using linear models, demonstrating robust performance across semantic relatedness, paraphrase detection, and image-sentence ranking.
Achieves comparable results to LSTM models trained specifically for semantic relatedness tasks, highlighting the effectiveness of unsupervised pre-training.
Demonstrates that combining skip-thoughts with bag-of-words features results in a strong baseline for text classification tasks.

Abstract

We describe an approach for unsupervised learning of a generic, distributed sentence encoder. Using the continuity of text from books, we train an encoder-decoder model that tries to reconstruct the surrounding sentences of an encoded passage. Sentences that share semantic and syntactic properties are thus mapped to similar vector representations. We next introduce a simple vocabulary expansion method to encode words that were not seen as part of training, allowing us to expand our vocabulary to a million words. After training our model, we extract and evaluate our vectors with linear models on 8 tasks: semantic relatedness, paraphrase detection, image-sentence ranking, question-type classification and 4 benchmark sentiment and subjectivity datasets. The end result is an off-the-shelf encoder that can produce highly generic sentence representations that are robust and perform well in practice. We will make our encoder publicly available.

Citation Graph

Loading graph...

References [42]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Very Deep Convolutional Networks for Large-Scale Image Recognition

K. Simonyan, Andrew Zisserman - 2014

20 papers in library cite

This is very good! The great thing here is small filters and depth analysis, but truly they do some other stuff as well: SotA, generalization for other tasks, and open source their models. Very nice.

[3]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[4]Microsoft COCO: Common Objects in Context

T. Y. Lin, M. Maire, S. Belongie, James Hays, Pietro Perona, D. Ramanan, Piotr Dollar, C. L. Zitnick - 2014

14 papers in library cite

I liked this paper a lot. It's a bit long and I was already a bit tired, but it was nice overall.

[5]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[6]Visualizing Data Using t-SNE

Geoffrey Hinton - 2008

7 papers in library cite

Amazing. Simple. Impactful. Easy to understand. Masterpiece.

[7]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[8]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[9]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[10]Convolutional Neural Networks for Sentence Classification

Yoon Kim - 2014

8 papers in library cite

It's nice, goes straight to the point. I can see why it has tons of citations. However, I am not sure it was as impactful as 20k citations.

[11]Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2014

11 papers in library cite

It's a good paper but results are veeeery underwhelming.

[12]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[13]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[14]On the Properties of Neural Machine Translation: Encoder-Decoder Approaches

Kyunghyun Cho, B. V. Merrienboer, D. Bahdanau, Yoshua Bengio - 2014

9 papers in library cite

This just builds upon other work, comparing and analyzing previous architectures. This faces a specific problem of the time: long sentences. Not too relevant today.

[15]A Convolutional Neural Network for Modelling Sentences

Phil Blunsom, Edward Grefenstette, N. Kalchbrenner - 2014

7 papers in library cite

I am usually biased against CNNs, but this was very good! Very well motivated and simple/intuitive explanations!

[16]Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books

Yuxuan Zhu, R. Kiros, R. Zemel, Ruslan Salakhutdinov, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

18 papers in library cite

I think their approach was a bit convoluted and didn't really add a lot. Main contribution here is probably BookCorpus

[17]Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks

Surya Ganguli - 2014

9 papers in library cite

TBH it's been almost 2 months since I read this paper (shame on me for forgetting to add it)... Anyway, as I recall it I liked it, but TBH it's a bit underwhelming because it solved only for linear networks

[18]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[19]Exploiting Similarities Among Languages for Machine Translation

Tomas Mikolov, Quoc V. Le, Ilya Sutskever - 2013

6 papers in library cite

It's just word2vec + a projection, but results are very nice and it's surprisingly simple!

[20]Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Richard Socher, Eric H. Huang, J. Pennin, C. Manning, A. Ng - 2011

10 papers in library cite

Good paper overall, but seems very simple ideas and complex implementation. Overall not very impactful

[21]Deep Visual-Semantic Alignments for Generating Image Descriptions

A. Karpathy, Li Fei Fei - 2014

6 papers in library cite

Early multimodality

[22]Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

K. S. Tai, Richard Socher, Christopher D. Manning - 2015

6 papers in library cite

Baseline for skip-thoughts paper

[23]Unsupervised Construction of Large Paraphrase corpora: Exploiting Massively Parallel News Sources

B. Dolan, C. Quirk, C. A. Brockett, C. Chris - 2004

5 papers in library cite

Seems very very close to the other paper I read, which I found boring. Probably not worth the read.

[24]Seeing Stars: Exploiting Class Relationships for Sentiment Categorization With Respect to Rating Scales

Bo Pang, L. Lee - 2005

13 papers in library cite

[25]A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts

Bo Pang, L. A. Lee, L. Lillian - 2004

8 papers in library cite

[26]Annotating Expressions of Opinions and Emotions in Language

J. Wiebe, T. Wilson, T. Theresa, C. A. Cardie, C. Claire - 2005

7 papers in library cite

[27]Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Shijie Wang, Manning, C. Christopher - 2012

7 papers in library cite

[28]Semeval-2014 Task 1: Evaluation of Compositional Distributional Semantic Models on Full Sentences Through Semantic Relatedness and Textual Entailment

Marco Marelli, L. Bentivogli, M. Baroni, R. Bernardi, S. Menini, R. Zamparelli - 2014

7 papers in library cite

[29]Mining and Summarizing Customer Reviews

M. Hu, B. A. Liu, B. Bing - 2004

6 papers in library cite

[30]Grounded Compositional Semantics for Finding and Describing Images With Sentences

Richard Socher, Quoc Le, C. Manning, A. Ng - 2014

5 papers in library cite

[31]Illinois-lh: A Denotational and Distributional Approach to Semantics

A. Lai, J. Hockenmaier - 2014

5 papers in library cite

[32]Deep Captioning With Multimodal Recurrent Neural Networks (M-Rnn)

J. Mao, Weixin Xu, Yining Yang, J. Wang, A. Yuille - 2014

3 papers in library cite

[33]Learning Question Classifiers

Xiang Lisa Li, Dan Roth - 2002

3 papers in library cite

[34]Paraphrase Identification as Probabilistic Quasi-Synchronous Recognition

Dipanjan Das, Noah A. Smith - 2009

3 papers in library cite

[35]Self-Adaptive Hierarchical Sentence Model

H. Zhao, Z. L. Lu, Z. A. Poupart, P. Pascal - 2015

3 papers in library cite

[36]Ecnu: One Stone Two Birds: Ensemble of Heterogenous Measures for Semantic Relatedness and Textual Entailment

J. Zhao, T. T. Zhu, M. Lan - 2014

2 papers in library cite

[37]Re-Examining Machine Translation Metrics for Paraphrase Identification

N. Madnani, J. Tetreault, J. A. Chodorow, M. M. K. Martin - 2012

2 papers in library cite

[38]Unal-Nlp: Combining Soft Cardinality Features for Semantic Textual Similarity, Relatedness and Entailment

S. Jimenez, G. Duenas, J. Baquero, A. Gelbukh, A. J. D. Batiz, A. Mendiz'abal - 2014

2 papers in library cite

[39]Using Dependency-Based Features to Take the "Para-Farce" Out of Paraphrase

S. Wan, M. Dras, R. Dale, C. Paris - 2006

2 papers in library cite

[40]Associating Neural Word Embeddings With Deep Image Representations Using Fisher Vectors

B. Klein, G. Lev, G. Sadeh, Lior Wolf - 2015

1 paper in library cites

[41]The Meaning Factory: Formal Semantics for Recognizing Textual Entailment and Determining Semantic Similarity

J. Bjerva, J. Bos, R. V. D. Goot, M. Nissim - 2014

1 paper in library cites

[42]Using Machine Translation Evaluation Techniques to Determine Sentence-Level Semantic Equivalence

A. Finch, Y. Hwang, E. Sumita - 2005

1 paper in library cites

Cited by

23

papers in your library

Cites

22

papers in your library

Read

on August 4, 2025

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

Tags

Paper Aliases

No aliases