Papperoni

2016

Learning Distributed Representations of Sentences From Unlabelled Data

F. Hill, Kyunghyun Cho, Anna Korhonen

Open PDF Google Scholar

citations

Cite Score

30

AI summary

This paper presents a systematic comparison of models for learning distributed sentence representations from unlabeled data, finding that deeper models perform better in supervised settings, while shallow log-linear models excel in unsupervised benchmarks, and introduces two new unsupervised representation-learning objectives to improve the trade-off between training time, domain portability and performance.

Main Contributions

Systematic comparison of sentence representation learning methods.
Propose Sequential Denoising Autoencoders, a new representation-learning objective.
Propose FastSent, a new representation-learning objective.
Deeper models generally perform best in the supervised setting.
Shallow log-linear models work best on unsupervised benchmarks.

Abstract

Unsupervised methods for learning distributed representations of words are ubiquitous in today's NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends critically on the intended application. Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-linear models work best for building representation spaces that can be decoded with simple spatial distance metrics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance.

Citation Graph

Loading graph...

References [43]

Sort:

Filter:

[1]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[2]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[3]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[4]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[5]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[6]Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov - 2014

13 papers in library cite

Introduced document embeddings. Very nice overall.

[7]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[8]Extracting and Composing Robust Features With Denoising Autoencoders

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre Antoine Manzagol - 2008

25 papers in library cite

I am *so* glad we found an alternative to DBNs. Also, introduced the idea of denoising which is nice.

[9]Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network With a Local Denoising Criterion

P. H. Vincent, Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre Antoine Manzagol - 2010

6 papers in library cite

This is basically a summary of everything that happened from 2006-2010, and also points some interesting things about DBNs! Very well explained as well.

[10]A Convolutional Neural Network for Modelling Sentences

Phil Blunsom, Edward Grefenstette, N. Kalchbrenner - 2014

7 papers in library cite

I am usually biased against CNNs, but this was very good! Very well motivated and simple/intuitive explanations!

[11]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[12]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[13]Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning - 2011

10 papers in library cite

Very nice and very simple method. I like this waaay better than the other Stanford recursive autoencoder papers.

[14]A Sick Cure for the Evaluation of Compositional Distributional Semantic Models

Marco Marelli, S. Menini, M. Baroni, L. Bentivogli, R. Bernardi, R. Z. Elli - 2014

7 papers in library cite

Just a basic dataset paper. Nothing much.

[15]Imagenet Large Scale Visual Recognition Challenge

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Zhongqiang Huang, A. Karpathy, A. Khosla, M. Bernstein - 2014

18 papers in library cite

Imagenet dataset challenge paper

[16]Neural Word Embeddings as Implicit Matrix Factorization

Omer Levy, Y. Goldberg - 2014

4 papers in library cite

Improve understanding of embeddings

[17]Building End-to-End Dialogue Systems Using Generative Hierarchical Neural Network Models

I. V. Serban, A. Sordoni, Yoshua Bengio, Aaron Courville, J. Pineau - 2016

2 papers in library cite

Dialogue systems - something that became very important after that

[18]Learning to Understand Phrases by Embedding the dictionary

F. Hill, Kyunghyun Cho, Anna Korhonen, Yoshua Bengio - 2015

1 paper in library cites

They embed dictionary definitions to get word meaning - nice

[19]Unsupervised Construction of Large Paraphrase corpora: Exploiting Massively Parallel News Sources

B. Dolan, C. Quirk, C. A. Brockett, C. Chris - 2004

5 papers in library cite

Seems very very close to the other paper I read, which I found boring. Probably not worth the read.

[20]Seeing Stars: Exploiting Class Relationships for Sentiment Categorization With Respect to Rating Scales

Bo Pang, L. Lee - 2005

13 papers in library cite

[21]A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts

Bo Pang, L. A. Lee, L. Lillian - 2004

8 papers in library cite

[22]Annotating Expressions of Opinions and Emotions in Language

J. Wiebe, T. Wilson, T. Theresa, C. A. Cardie, C. Claire - 2005

7 papers in library cite

[23]Mining and Summarizing Customer Reviews

M. Hu, B. A. Liu, B. Bing - 2004

6 papers in library cite

[24]Composition in Distributional Models of Semantics

J. Mitchell, Mirella Lapata - 2010

5 papers in library cite

[25]SemEval-2014 Task 10: Multilingual Semantic Textual Similarity

E. Agirre, C. Banea, C. Cardie, D. M. Cer, M. T. Diab, A. G. Agirre, W. Guo, R. Mihalcea, G. Rigau, J. Wiebe - 2014

4 papers in library cite

[26]Deep Captioning With Multimodal Recurrent Neural Networks (M-Rnn)

J. Mao, Weixin Xu, Yining Yang, J. Wang, A. Yuille - 2014

3 papers in library cite

[27]Discriminative Improvements to Distributional Sentence Similarity

Yangfeng Ji, J. Eisenstein - 2013

3 papers in library cite

[28]Distributional Structure

Z. Harris - 1954

3 papers in library cite

[29]Don't Count, Predict! A Systematic Comparison of Context-Counting vs Context-Predicting Semantic Vectors

M. Baroni, G. Dinu, German Kruszewski - 2014

3 papers in library cite

[30]Simlex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

F. Hill, R. Reichart, Anna Korhonen - 2015

3 papers in library cite

[31]Vector-Based Models of Semantic Composition

J. Mitchell, Mirella Lapata - 2008

3 papers in library cite

[32]Combining Symbolic and Distributional Models of Meaning

S. Clark, S. Pulman - 2007

2 papers in library cite

[33]Jointly Optimizing Word Representations for Lexical and Sentential Tasks With the C-Phrase Model

N. T. Pham, German Kruszewski, A. Lazaridou, M. Baroni - 2015

2 papers in library cite

[34]Multimodal Distributional Semantics

E. Bruni, N. K. Tran, M. Baroni - 2014

2 papers in library cite

[35]An Exploration of Discourse-Based Sentence Spaces for Compositional Distributional Semantics

T. Polajnar, L. Rimell, S. Clark - 2015

1 paper in library cites

[36]Deep Unordered Composition Rivals Syntactic Methods for Text Classification

M. Iyyer, V. Manjunatha, J. B. Graber, H. D. Iii - 2015

1 paper in library cites

[37]Evaluating Neural Word Representations in Tensor-Based Compositional Settings

D. Milajevs, D. Kartsaklis, M. Sadrzadeh, M. Purver - 2014

1 paper in library cites

[38]Frege in Space: A Program of Compositional Distributional Semantics

M. Baroni, R. Bernardi, R. Zamparelli - 2014

1 paper in library cites

[39]Improving Distributional Similarity With Lessons Learned From Word Embeddings

Omer Levy, Y. Goldberg, Ido Dagan - 2015

1 paper in library cites

[40]Learning Distributed Representations From Reviews for Collaborative Filtering

Amjad Almahairi, K. Kastner, Kyunghyun Cho, Aaron Courville - 2015

1 paper in library cites

[41]Memory, Knowledge, and the Answering of Questions

D. A. Norman - 1972

1 paper in library cites

[42]Microsoft Coco Captions: Data Collection and Evaluation Server

X. Chen, H. Fang, T. Y. Lin, R. Vedantam, S. Gupta, Piotr Dollar, C. L. Zitnick - 2015

1 paper in library cites

[43]Overview of the Trec 2001 Question Answering Track

E. M. Voorhees - 2002

1 paper in library cites

Cited by

12

papers in your library

Cites

18

papers in your library

Read

on October 27, 2025

I liked it overall (almost gave it 4 starts), but I am a bit tired of embeddings. It's better than the other ones though. FastSent is nice.

Tags

Paper Aliases

No aliases