Papperoni

2012

Improving Word Representations via Global Context and Multiple Word Prototypes

Eric H. Huang, Richard Socher, C. Manning, Andrew Y. Ng

Open PDF Google Scholar

citations

Cite Score

49

AI summary

This paper introduces a novel neural network architecture for learning improved word embeddings by leveraging both local and global document context, addressing homonymy and polysemy through multiple embeddings per word. The model achieves superior performance on a new dataset of human similarity judgments in sentential context.

Main Contributions

Introduces a new neural network architecture that learns word embeddings by incorporating both local and global document context.
Accounts for homonymy and polysemy by learning multiple embeddings per word.
Introduces a new dataset with human judgments on pairs of words in sentential context.
Demonstrates that the model outperforms competitive baselines and other neural language models.
Shows that the multi-prototype approach can improve over the single-prototype version.

Abstract

Unsupervised word representations are very useful in NLP tasks both as inputs to learning algorithms and as extra word features in NLP systems. However, most of these models are built with only local context and one representation per word. This is problematic because words are often polysemous and global context can also provide useful information for learning word meanings. We present a new neural network architecture which 1) learns word embeddings that better capture the semantics of words by incorporating both local and global document context, and 2) accounts for homonymy and polysemy by learning multiple embeddings per word. We introduce a new dataset with human judgments on pairs of words in sentential context, and evaluate our model on it, showing that our model outperforms competitive baselines and other neural language models.

Citation Graph

Loading graph...

References [37]

Sort:

Filter:

[1]WordNet: A Lexical Database for English

G. Miller - 1995

5 papers in library cite

Meh. It seems like it was publish so that people could cite the dataset somehow. Nothing interesting, but quick read and very used.

[2]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[3]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[4]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

This basically introduced the concept of embeddings. Very nice.

[5]Parsing Natural Scenes and Natural Language With Recursive Neural Networks

Richard Socher, C. C. Lin, C. Manning, Andrew Y. Ng - 2011

10 papers in library cite

Good idea and nice results, but not my thing right now.

[6]Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning - 2011

10 papers in library cite

Very nice and very simple method. I like this waaay better than the other Stanford recursive autoencoder papers.

[7]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[8]Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Richard Socher, Eric H. Huang, J. Pennin, C. Manning, A. Ng - 2011

10 papers in library cite

Good paper overall, but seems very simple ideas and complex implementation. Overall not very impactful

[9]Three New Graphical Models for Statistical Language Modelling

A. Mnih, Geoffrey Hinton - 2007

12 papers in library cite

I don't know why this is so impactful. I didn't like it and I think this was overly complex.

[10]Connectionist Language Modeling for Large Vocabulary Continuous Speech Recognition

Holger Schwenk, Jean Luc Gauvain - 2002

14 papers in library cite

Only real relevance is being early. Otherwise not much to see.

[11]Using a Connectionist Model in a Syntactical Based Language Model

Frederick Jelinek - 2003

6 papers in library cite

A copy of the other paper. I still don't get what the "Structured Language Model" is and this seems like trying to fit a cube in a circular hole. I don't like it.

[12]A Structured Vector Space Model for Word Meaning in Context

K. Erk, S. Pado - 2008

3 papers in library cite

[13]Contextual Correlates of Semantic Similarity

George A. Miller, W. G. Charles - 1991

3 papers in library cite

[14]Cheap and Fast-but Is It Good?: Evaluating Non-Expert Annotations for Natural Language Tasks

R. Snow, B. O'connor, Dan Jurafsky, Andrew Y. Ng - 2008

2 papers in library cite

[15]Contextual Correlates of Synonymy

H. Rubenstein, J. B. Goodenough - 1965

2 papers in library cite

[16]Introduction to Information Retrieval

Christopher D. Manning, P. Raghavan, H. Schtze - 2008

2 papers in library cite

[17]Machine Learning in Automated Text Categorization

F. Sebastiani - 2002

2 papers in library cite

[18]Multi-Prototype Vector-Space Models of Word Meaning

J. Reisinger, R. J. Mooney - 2010

2 papers in library cite

[19]On the Limited Memory BFGS Method for Large Scale Optimization

D. C. Liu, J. Nocedal - 1989

2 papers in library cite

[20]Placing Search in Context: The Concept Revisited

L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, E. Ruppin - 2001

2 papers in library cite

[21]Quantitative Evaluation of Passage Retrieval Algorithms for Question Answering

S. Tellex, B. Katz, Junyang Lin, A. Fernandes, G. Marton - 2003

2 papers in library cite

[22]The Westbury Lab Wikipedia Corpus

C. Shaoul, C. Westbury - 2010

2 papers in library cite

[23]A Mixture Model With Sharing for Lexical Semantics

J. Reisinger, R. Mooney - 2010

1 paper in library cites

[24]Automatic Word Sense Discrimination

Hinrich Schutze - 1998

1 paper in library cites

[25]Computing Semantic Relatedness Using Wikipedia-Based Explicit Semantic Analysis

E. Gabrilovich, S. Markovitch - 2007

1 paper in library cites

[26]Concept Decompositions for Large Sparse Text Data Using Clustering

I. S. Dhillon, D. S. Modha - 2001

1 paper in library cites

[27]Corpus-Based Approaches to Semantic Interpretation in Natural Language Processing

H. Ng, J. Zelle - 1997

1 paper in library cites

[28]Dynamic and Static Prototype Vectors for Semantic Composition

Siva Reddy, I. Klapaftis, D. Mccarthy, S. Manandhar - 2011

1 paper in library cites

[29]Effects of Global and Local Context on Lexical Processing During Language Comprehension

D. J. Hess, D. J. Foss, P. Carroll - 1995

1 paper in library cites

[30]From Distributional to Semantic Similarity

J. R. Curran - 2004

1 paper in library cites

[31]Measuring Distributional Similarity in Context

G. Dinu, Mirella Lapata - 2010

1 paper in library cites

[32]Mixture Models of Categorization

Y. Rosseel - 2002

1 paper in library cites

[33]Multi-View Learning of Word Embeddings via Cca

P. S. Dhillon, D. Foster, L. Ungar - 2011

1 paper in library cites

[34]Semeval-2010 Task 14: Word Sense Induction & Disambiguation

S. Manandhar, I. P. Klapaftis, D. Dligach, S. S. Pradhan - 2010

1 paper in library cites

[35]The Acquisition of Word Meaning Through Global Lexical Co-Occurrences

P. L. Li, C. Burgess, K. Lund - 2000

1 paper in library cites

[36]Unifying Rational Models of Categorization via the Hierarchical Dirichlet Process

T. L. Griffiths, K. R. Canini, A. N. Sanborn, D. J. Navarro - 2009

1 paper in library cites

[37]Word Meaning in Context: A Simple and Effective Vector Model

S. Thater, H. Furstenau, M. Pinkal - 2011

1 paper in library cites

Cited by

7

papers in your library

Cites

11

papers in your library

Read

on March 20, 2025

Good paper. I would say not too relevant, but has some nice concepts like a "global context vector" (document embedding) that comes from averaging word embeddings, and a joint training objective.

Tags

Paper Aliases

No aliases