Papperoni

2014

Distributed Representations of Sentences and Documents

Quoc Le, Tomas Mikolov

Open PDF Google Scholar

citations

Cite Score

88

AI summary

This paper introduces Paragraph Vector, an unsupervised algorithm using distributed memory and distributed bag of words to learn fixed-length feature representations from variable-length texts. The method achieves state-of-the-art results on several text classification and sentiment analysis tasks.

Main Contributions

Introduces Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length texts.
Proposes Distributed Memory Model of Paragraph Vectors (PV-DM) and Distributed Bag of Words version of Paragraph Vector (PV-DBOW).
Achieves state-of-the-art results on sentiment analysis tasks, outperforming bag-of-words models and recursive neural networks on the Stanford Sentiment Treebank dataset.
Demonstrates strong performance on the IMDB dataset for sentiment analysis, surpassing previous methods.
Shows effectiveness on an information retrieval task, achieving a 32% relative improvement in error rate compared to bag-of-words and bigrams.

Abstract

Many machine learning algorithms require the input to be represented as a fixed-length feature vector. When it comes to texts, one of the most common fixed-length features is bag-of-words. Despite their popularity, bag-of-words features have two major weaknesses: they lose the ordering of the words and they also ignore semantics of the words. For example, “powerful,” “strong” and "Paris" are equally distant. In this paper, we propose Paragraph Vector, an unsupervised algorithm that learns fixed-length feature representations from variable-length pieces of texts, such as sentences, paragraphs, and documents. Our algorithm represents each document by a dense vector which is trained to predict words in the document. Its construction gives our algorithm the potential to overcome the weaknesses of bag-of-words models. Empirical results show that Paragraph Vectors outperform bag-of-words models as well as other techniques for text representations. Finally, we achieve new state-of-the-art results on several text classification and sentiment analysis tasks.

Citation Graph

Loading graph...

References [38]

Sort:

Filter:

[1]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[2]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[3]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[4]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[5]Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

Richard Socher, A. Perelygin, Jeffrey Wu, J. Chuang, C. Manning, A. Ng, Christopher Potts - 2013

24 papers in library cite

I didn't really like the first paper and I don't really like this one. I think the dataset is more influential than the methodology. I think Stanford folks are too focused on old school NLP.

[6]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[7]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[8]Learning Word Vectors for Sentiment Analysis

A. L. Maas, R. E. Daly, P. T. Pham, Dong Huang, Andrew Y. Ng, Christopher Potts - 2011

12 papers in library cite

Very convoluted method. I did not like it and I had to go back twice to remember if I had read it or not. Main contribution is the IMDB movie review dataset. (It is cited by Mikolov as a comparison)

[9]Linguistic Regularities in Continuous Space Word Representations

Tomas Mikolov, W. T. Yih, Geoffrey Zweig - 2013

8 papers in library cite

This builds on Bengio's "Word Representations" paper to show that embeddings can be used to describe syntactic and semantic characteristics of words. Good contribution, but not sure if too impactful.

[10]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

This basically introduced the concept of embeddings. Very nice.

[11]Exploiting Similarities Among Languages for Machine Translation

Tomas Mikolov, Quoc V. Le, Ilya Sutskever - 2013

6 papers in library cite

It's just word2vec + a projection, but results are very nice and it's surprisingly simple!

[12]Parsing Natural Scenes and Natural Language With Recursive Neural Networks

Richard Socher, C. C. Lin, C. Manning, Andrew Y. Ng - 2011

10 papers in library cite

Good idea and nice results, but not my thing right now.

[13]Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning - 2011

10 papers in library cite

Very nice and very simple method. I like this waaay better than the other Stanford recursive autoencoder papers.

[14]Improving Word Representations via Global Context and Multiple Word Prototypes

Eric H. Huang, Richard Socher, C. Manning, Andrew Y. Ng - 2012

7 papers in library cite

Good paper. I would say not too relevant, but has some nice concepts like a "global context vector" (document embedding) that comes from averaging word embeddings, and a joint training objective.

[15]Hierarchical Probabilistic Neural Network Language Model

F. Morin, Yoshua Bengio - 2005

19 papers in library cite

Nice paper overall. Seems very impactful despite not being as relevant right now.

[16]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[17]Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Richard Socher, Eric H. Huang, J. Pennin, C. Manning, A. Ng - 2011

10 papers in library cite

Good paper overall, but seems very simple ideas and complex implementation. Overall not very impactful

[18]Devise: A Deep Visual-Semantic Embedding Model

Andrea Frome, G. S. Corrado, J. Shlens, Samy Bengio, Jeffrey Dean, Tomas Mikolov, Marc'aurelio Ranzato - 2013

4 papers in library cite

Identify objects

[19]Bilingual Word Embeddings for Phrase-Based Machine Translation

W. Zou, Richard Socher, D. Cer, C. Manning - 2013

4 papers in library cite

Bilingual embeddings?

[20]Combining Heterogeneous Models for Measuring Relational Similarity

A. Zhila, W. T. Yih, C. Meek, Geoffrey Zweig, Tomas Mikolov - 2013

2 papers in library cite

[21]Statistical Language Models Based on Neural Networks

Tomas Mikolov - 2012

17 papers in library cite

Mikolov's Thesis

[22]Seeing Stars: Exploiting Class Relationships for Sentiment Categorization With Respect to Rating Scales

Bo Pang, L. Lee - 2005

13 papers in library cite

[23]Accurate Unlexicalized Parsing

Dan Klein, Christopher D. Manning - 2003

7 papers in library cite

[24]Baselines and Bigrams: Simple, Good Sentiment and Topic Classification

Shijie Wang, Manning, C. Christopher - 2012

7 papers in library cite

[25]From Frequency to Meaning: Vector Space Models of Semantics

P. D. Turney, P. Pantel - 2010

6 papers in library cite

[26]Composition in Distributional Models of Semantics

J. Mitchell, Mirella Lapata - 2010

5 papers in library cite

[27]Distributional Structure

Z. Harris - 1954

3 papers in library cite

[28]Estimating Linear Models for Compositional Distributional Semantics

F. Zanzotto, I. Korkontzelos, F. Fallucchi, S. Manandhar - 2010

3 papers in library cite

[29]Exploiting Generative Models in Discriminative Classifiers

T. Jaakkola, D. Haussler - 1999

3 papers in library cite

[30]Fisher Kernels on Visual Vocabularies for Image Categorization

F. Perronnin, C. Dance - 2007

3 papers in library cite

[31]Neural Probabilistic Language Models

Yoshua Bengio, Holger Schwenk, Jean Sebastien Senecal, F. Morin, Jean Luc Gauvain - 2006

3 papers in library cite

[32]Compositional Matrix-Space Models for Sentiment Analysis

A. Yessenalina, C. Cardie - 2011

2 papers in library cite

[33]Multi-Step Regression Learning for Compositional Distributional Semantics

Edward Grefenstette, G. Dinu, Y. Z. Zhang, M. Sadrzadeh, M. Baroni - 2013

2 papers in library cite

[34]Reasoning With Neural Tensor Networks for Knowledge Base Completion

Richard Socher, Deli Chen, Christopher D. Manning, A. Ng - 2013

2 papers in library cite

[35]A Neural Autoregressive Topic Model

Hugo Larochelle, S. Lauly - 2012

1 paper in library cites

[36]Large-Scale Image Retrieval With Compressed Fisher vectors

F. Perronnin, Yibo Liu, J. Sanchez, H. Poirier - 2010

1 paper in library cites

[37]Modeling Documents With Deep Boltzmann Machines

N. Srivastava, Ruslan Salakhutdinov, Geoffrey Hinton - 2013

1 paper in library cites

[38]Training Restricted Boltzmann Machines on Word Observations

George E. Dahl, R. P. Adams, Hugo Larochelle - 2012

1 paper in library cites

Cited by

13

papers in your library

Cites

20

papers in your library

Read

on April 21, 2025

Introduced document embeddings. Very nice overall.

Tags

Paper Aliases

No aliases