Papperoni

2014

GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning

Open PDF Google Scholar

citations

Cite Score

97

AI summary

This paper introduces GloVe, a global log-bilinear regression model that combines global matrix factorization and local context window methods for learning word vector representations, achieving state-of-the-art results on word analogy datasets.

Main Contributions

Proposed GloVe, a new global log-bilinear regression model for unsupervised word representation learning.
The model combines global matrix factorization and local context window methods.
Achieved state-of-the-art performance of 75% accuracy on the word analogy dataset.
Outperformed related models on word similarity tasks and named entity recognition.
Analyzed model properties to produce linear directions of meaning

Abstract

Recent methods for learning vector space representations of words have succeeded in capturing fine-grained semantic and syntactic regularities using vector arithmetic, but the origin of these regularities has remained opaque. We analyze and make explicit the model properties needed for such regularities to emerge in word vectors. The result is a new global log-bilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods. Our model efficiently leverages statistical information by training only on the nonzero elements in a word-word co-occurrence matrix, rather than on the entire sparse matrix or on individual context windows in a large corpus. The model produces a vector space with meaningful sub-structure, as evidenced by its performance of 75% on a recent word analogy task. It also outperforms related models on similarity tasks and named entity recognition.

Citation Graph

Loading graph...

References [29]

Sort:

Filter:

[1]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[2]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[3]Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

John Duchi, Elad Hazan, Yoram Singer - 2011

19 papers in library cite

I actually skimmed through most of this. It's not a bad paper, but it's a math paper, not AI.

[4]Learning Deep Architectures for AI

Yoshua Bengio - 2009

25 papers in library cite

It's a nice overview. Some sections get very theoretical, but the first half is very good and I feel that it does a waaaay better job of explaining RBMs and DBNs than other papers. This feels like Bengio is taking your hand and saying "if you don't know what's going on, here you go, everything you need to know to jump into the deep nets train"

[5]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[6]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[7]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[8]Linguistic Regularities in Continuous Space Word Representations

Tomas Mikolov, W. T. Yih, Geoffrey Zweig - 2013

8 papers in library cite

This builds on Bengio's "Word Representations" paper to show that embeddings can be used to describe syntactic and semantic characteristics of words. Good contribution, but not sure if too impactful.

[9]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

This basically introduced the concept of embeddings. Very nice.

[10]Improving Word Representations via Global Context and Multiple Word Prototypes

Eric H. Huang, Richard Socher, C. Manning, Andrew Y. Ng - 2012

7 papers in library cite

Good paper. I would say not too relevant, but has some nice concepts like a "global context vector" (document embedding) that comes from averaging word embeddings, and a joint training objective.

[11]Indexing by Latent Semantic Analysis

S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, R. Harshman - 1990

12 papers in library cite

LSA paper

[12]Better Word Representations With Recursive Neural Networks for Morphology

T. Luong, Richard Socher, Christopher D. Manning - 2013

4 papers in library cite

[13]Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition

E. F. T. K. Sang, F. D. Meulder - 2003

4 papers in library cite

[14]Learning Word Embeddings Efficiently With Noise-Contrastive Estimation

A. Mnih, Koray Kavukcuoglu - 2013

4 papers in library cite

[15]Contextual Correlates of Semantic Similarity

George A. Miller, W. G. Charles - 1991

3 papers in library cite

[16]Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images

Dan C. Ciresan, A. Giusti, Jürgen Schmidhuber - 2012

3 papers in library cite

[17]Don't Count, Predict! A Systematic Comparison of Context-Counting vs Context-Predicting Semantic Vectors

M. Baroni, G. Dinu, German Kruszewski - 2014

3 papers in library cite

[18]Parsing With Compositional Vector Grammars

Richard Socher, J. Bauer, Christopher D. Manning, Andrew Y. Ng - 2013

3 papers in library cite

[19]Producing High-Dimensional Semantic Spaces From Lexical Co-Occurrence

K. Lund, C. Burgess - 1996

3 papers in library cite

[20]Contextual Correlates of Synonymy

H. Rubenstein, J. B. Goodenough - 1965

2 papers in library cite

[21]Machine Learning in Automated Text Categorization

F. Sebastiani - 2002

2 papers in library cite

[22]Placing Search in Context: The Concept Revisited

L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, E. Ruppin - 2001

2 papers in library cite

[23]Quantitative Evaluation of Passage Retrieval Algorithms for Question Answering

S. Tellex, B. Katz, Junyang Lin, A. Fernandes, G. Marton - 2003

2 papers in library cite

[24]An Improved Model of Semantic Similarity Based on Lexical Co-Occurence

D. L. T. Rohde, L. M. Gonnerman, D. C. Plaut - 2006

1 paper in library cites

[25]Effect of Non-Linear Deep Architecture in Sequence Labeling

Mingliang Wang, Christopher D. Manning - 2013

1 paper in library cites

[26]Extracting Semantic Representations From Word Co-Occurrence Statistics: A Computational Study

J. A. Bullinaria, J. P. Levy - 2007

1 paper in library cites

[27]Introduction to Analytic Number Theory

T. M. Apostol - 1976

1 paper in library cites

[28]Linguistic Regularities in Sparse and Explicit Word Representations

Omer Levy, Y. Goldberg, I. R. Gan - 2014

1 paper in library cites

[29]Word Embeddings Through Hellinger PCA

R. Lebret, Ronan Collobert - 2014

1 paper in library cites

Cited by

31

papers in your library

Cites

11

papers in your library

Read

on August 8, 2025

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

Tags

Paper Aliases

No aliases