Papperoni

2016

context2vec: Learning Generic Context Embedding With Bidirectional LSTM

O. Melamud, J. Goldberger, Ido Dagan

Open PDF Google Scholar

citations

Cite Score

29

AI summary

This paper introduces context2vec, a bidirectional LSTM model, to learn generic context embeddings from large corpora; it achieves state-of-the-art results on sentence completion, lexical substitution, and word sense disambiguation tasks and outperforms averaged word embeddings.

Main Contributions

Introduces context2vec, a novel neural network architecture based on bidirectional LSTMs for learning generic context embeddings.
Demonstrates that context2vec can be trained efficiently on large corpora.
Shows that context2vec achieves state-of-the-art or near state-of-the-art results on sentence completion, lexical substitution, and word sense disambiguation tasks.
Releases code and pre-trained models for context2vec.

Abstract

Context representations are central to various NLP tasks, such as word sense disambiguation, named entity recognition, co-reference resolution, and many more. In this work we present a neural model for efficiently learning a generic context embedding function from large corpora, using bidirectional LSTM. With a very simple application of our context representations, we manage to surpass or nearly reach state-of-the-art results on sentence completion, lexical substitution and word sense disambiguation tasks, while substantially outperforming the popular context representation of averaged word embeddings. We release our code and pre-trained models, suggesting they could be useful in a wide variety of NLP tasks.

Citation Graph

Loading graph...

References [36]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[3]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[4]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[5]Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa - 2011

23 papers in library cite

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

[6]Skip-Thought Vectors

R. Kiros, Yuxuan Zhu, Ruslan Salakhutdinov, Richard S. Zemel, R. Urtasun, Antonio Torralba, Sanja Fidler - 2015

23 papers in library cite

Nice to see an alternative to Word2Vec to sentences, but I don't really like the approach. Good nonetheless.

[7]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

This basically introduced the concept of embeddings. Very nice.

[8]Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions

Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, Christopher D. Manning - 2011

10 papers in library cite

Very nice and very simple method. I like this waaay better than the other Stanford recursive autoencoder papers.

[9]Improving Word Representations via Global Context and Multiple Word Prototypes

Eric H. Huang, Richard Socher, C. Manning, Andrew Y. Ng - 2012

7 papers in library cite

Good paper. I would say not too relevant, but has some nice concepts like a "global context vector" (document embedding) that comes from averaging word embeddings, and a joint training objective.

[10]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[11]Neural Word Embeddings as Implicit Matrix Factorization

Omer Levy, Y. Goldberg - 2014

4 papers in library cite

Improve understanding of embeddings

[12]The microsoft research Sentence Completion Challenge

Geoffrey Zweig, C. J. Burges - 2011

6 papers in library cite

[13]Neural Architectures for Named Entity Recognition

G. Lample, M. Ballesteros, S. Subramanian, K. Kawakami, C. Dyer - 2016

4 papers in library cite

[14]End-to-End Learning of Semantic Role Labeling Using Recurrent Neural Networks

Jingren Zhou, Weixin Xu - 2015

3 papers in library cite

[15]Not All Contexts Are Created Equal: Better Word Representations With Variable Attention

W. Ling, L. C. Cheng, Y. Tsvetkov, S. Amir - 2015

2 papers in library cite

[16]A Simple Word Embedding Model for Lexical Substitution

O. Melamud, Omer Levy, Ido Dagan - 2015

1 paper in library cites

[17]A Unified Model for Word Sense Representation and Disambiguation

X. Chen, Ze Liu, Maosong Sun - 2014

1 paper in library cites

[18]Applying Alternating Structure Optimization to Word Sense Disambiguation

Rie Kubota Ando - 2006

1 paper in library cites

[19]Autoextend: Extending Word Embeddings to Embeddings for Synsets and Lexemes

S. Rothe, Hinrich Schutze - 2015

1 paper in library cites

[20]Chainer: A Next-Generation Open Source Framework for Deep Learning

S. Tokui, K. Oono, S. Hido, J. Clayton - 2015

1 paper in library cites

[21]Dependency Recurrent Neural Language Models for Sentence Completion

P. Mirowski, A. Vlachos - 2015

1 paper in library cites

[22]Dependency-Based Word Embeddings

Omer Levy, Y. Goldberg - 2014

1 paper in library cites

[23]FASTSUBS: An Efficient and Exact Procedure for Finding the Most Likely Lexical Substitutes Based on an N-Gram Language Model

Deniz Yuret - 2012

1 paper in library cites

[24]Introducing and Evaluating Ukwac, a Very Large Web-Derived Corpus of English

A. Ferraresi, E. Zanchetta, M. Baroni, S. Bernardini - 2008

1 paper in library cites

[25]Learning Semantic Word Embeddings Based on Ordinal Knowledge Constraints

Qian Liu, H. Jiang, S. Wei, Z. H. Ling, Y. Hu - 2015

1 paper in library cites

[26]Learning Syntactic Categories Using Paradigmatic Representations of Word Context

E. Sert, Deniz Yuret - 2012

1 paper in library cites

[27]Learning to Rank Lexical Substitutions

G. Szarvas, R. B. Fekete, Eyke Hüllermeier - 2013

1 paper in library cites

[28]Learning to Represent Words in Context With Multilingual Supervision

K. Kawakami, C. Dyer - 2016

1 paper in library cites

[29]Modeling Word Meaning in Context With Substitute Vectors

O. Melamud, Ido Dagan, J. Goldberger - 2015

1 paper in library cites

[30]Neural Context Embeddings for Automatic Discovery of Word Senses

M. Kageback, F. Johansson, R. Johansson, D. Dubhashi - 2015

1 paper in library cites

[31]Semeval-2007 Task 10: English Lexical Substitution Task

D. Mccarthy, R. Navigli - 2007

1 paper in library cites

[32]Tailoring Continuous Word Representations for Dependency Parsing

Mohit Bansal, Kevin Gimpel, Karen Livescu - 2014

1 paper in library cites

[33]The Role of Context Types and Dimensionality in Learning Word Embeddings

O. Melamud, D. Mcclosky, S. Patwardhan, Mohit Bansal - 2016

1 paper in library cites

[34]The Senseval-3 English Lexical Sample Task

R. Mihalcea, T. A. Chklovski, A. Kilgarriff - 2004

1 paper in library cites

[35]Two/Too Simple Adaptations of Word2vec for Syntax Problems

W. Ling, C. Dyer, A. Black, I. Trancoso - 2015

1 paper in library cites

[36]What Substitutes Tell Us-Analysis of an All-Words Lexical Substitution Corpus

G. Kremer, K. Erk, S. Pado, S. Thater - 2014

1 paper in library cites

Cited by

5

papers in your library

Cites

11

papers in your library

Read

on October 27, 2025

I liked the methodology a lot but the paper reads like an MSc course project.

Tags

Paper Aliases

No aliases