Papperoni

2010

Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

63

AI summary

This paper introduces a simple way to improve NLP accuracy by using unsupervised word representations (Brown clusters, Collobert & Weston embeddings, and HLBL embeddings) as extra word features, achieving improved results on NER and chunking tasks.

Main Contributions

Demonstrates that unsupervised word representations can improve the accuracy of existing supervised NLP systems.
Evaluates Brown clusters, Collobert and Weston embeddings, and HLBL embeddings on NER and chunking tasks.
Shows that combining different word representations can lead to further improvements in accuracy.
Provides a method for scaling word embeddings for off-the-shelf use as word features.
Offers a default method for setting the scaling parameter for word embeddings

Abstract

If we take an existing supervised NLP system, a simple and general way to improve accuracy is to use unsupervised word representations as extra word features. We evaluate Brown clusters, Collobert and Weston (2008) embeddings, and HLBL (Mnih & Hinton, 2009) embeddings of words on both NER and chunking. We use near state-of-the-art supervised baselines, and find that each of the three word representations improves the accuracy of these baselines. We find further improvements by combining different word representations. You can download our word features, for off-the-shelf use in existing NLP systems, as well as our code, here: http://metaoptimize.com/projects/wordreprs/

Citation Graph

Loading graph...

References [50]

Sort:

Filter:

[1]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[2]A Unified Architecture for Natural Language Processing: Deep Neural Networks With Multitask Learning

Ronan Collobert, Jason Weston - 2008

32 papers in library cite

Really did not add much to the game. I think this was more of a small perf. improvement over other existing things and set a few methodological standards. Maybe main contribution is Multitask Learning + Deep learning

[3]Curriculum Learning

Yoshua Bengio, J. Louradour, Ronan Collobert, Jason Weston - 2009

6 papers in library cite

Very nice paper that introduces curriculum learning. Possibly not too relevant, but good nonetheless.

[4]Learning and Development in Neural Networks: The Importance of Starting Small

Jeffrey L. Elman - 1993

5 papers in library cite

This is such a nice paper! Maybe because it's written for a specific public, but it's such an easy read and ties back a lot with neural/biological concepts!

[5]Hierarchical Probabilistic Neural Network Language Model

F. Morin, Yoshua Bengio - 2005

19 papers in library cite

Nice paper overall. Seems very impactful despite not being as relevant right now.

[6]A Scalable Hierarchical Distributed Language Model

A. Mnih, Geoffrey E. Hinton - 2009

16 papers in library cite

Good paper that introduces hierarchical trees as an alternative to the expensive softmax output. I think this is not really relevant anymore, but good read.

[7]Three New Graphical Models for Statistical Language Modelling

A. Mnih, Geoffrey Hinton - 2007

12 papers in library cite

I don't know why this is so impactful. I didn't like it and I think this was overly complex.

[8]Quick Training of Probabilistic Neural Nets by Importance Sampling

Yoshua Bengio, Jean Sebastien Senecal - 2003

11 papers in library cite

Good idea to overcome softmax computation cost. Not sure if too relevant today, but definitely better than the 2008 paper that is the same stuff.

[9]Connectionist Language Modeling for Large Vocabulary Continuous Speech Recognition

Holger Schwenk, Jean Luc Gauvain - 2002

14 papers in library cite

Only real relevance is being early. Otherwise not much to see.

Reference title contains 'et al'

[10]Latent Dirichlet Allocation

D. M. Blei, Andrew Y. Ng, Michael I. Jordan - 2003

10 papers in library cite

30 pages; LDA

[11]Class-Based N-Gram Models of Natural Language

P. F. Brown, P. V. Desouza, R. L. Mercer, Vincent J. Della Pietra, J. C. Lai - 1992

12 papers in library cite

[12]From Frequency to Meaning: Vector Space Models of Semantics

P. D. Turney, P. Pantel - 2010

6 papers in library cite

[13]A High-Performance Semi-Supervised Learning Method for Text Chunking

Rie Kubota Ando, Tong Zhang - 2005

4 papers in library cite

[14]Distributional Clustering of English Words

Fernando Pereira, N. Tishby, L. Lee - 1993

4 papers in library cite

[15]Semi-Supervised Sequential Labeling and Segmentation Using Gigaword Scale Unlabeled Data

J. Suzuki, H. Isozaki - 2008

4 papers in library cite

[16]Design Challenges and Misconceptions in Named Entity Recognition

L. A. Ratinov, Dan Roth - 2009

3 papers in library cite

[17]Producing High-Dimensional Semantic Spaces From Lexical Co-Occurrence

K. Lund, C. Burgess - 1996

3 papers in library cite

[18]A Robust Risk Minimization Based Named Entity Recognition System

Tong Zhang, D. Johnson - 2003

2 papers in library cite

[19]Distributional Representations for Handling Sparsity in Supervised Sequence-Labeling

F. Huang, A. Yates - 2009

2 papers in library cite

[20]Introduction to the CoNLL-2000 Shared Task Chunking

E. F. T. K. Sang, S. Buchholz - 2000

2 papers in library cite

[21]Name Tagging With Word Clusters and Discriminative Training

S. Miller, J. Guinness, A. Zamanian - 2004

2 papers in library cite

[22]Neural Net Language Models

Yoshua Bengio - 2008

2 papers in library cite

[23]Phrase Clustering for Discriminative Learning

D. Lin, Xiaobao Wu - 2009

2 papers in library cite

[24]Semi-Supervised Sequence Modeling With Syntactic Topic Models

Wentao Li, Andrew Mccallum - 2005

2 papers in library cite

[25]Shallow Parsing With Conditional Random Fields

F. Sha, Fernando Pereira - 2003

2 papers in library cite

[26]Simple Semi-Supervised Dependency Parsing

T. Koo, X. Carreras, Michael Collins - 2008

2 papers in library cite

[27]A Preliminary Evaluation of Word Representations for Named-Entity Recognition

J. Turian, L. Ratinov, Yoshua Bengio, Dan Roth - 2009

1 paper in library cites

[28]Algorithms for bigram and trigram Word Clustering

S. Martin, J. Liermann, Hermann Ney - 1998

1 paper in library cites

[29]An Effective Two-Stage Model for Exploiting Non-Local Dependencies in Named Entity Recognition

V. Krishnan, Christopher D. Manning - 2006

1 paper in library cites

[30]An Empirical Study of Semi-Supervised Structured Conditional Models for Dependency Parsing

J. Suzuki, H. Isozaki, X. Carreras, Michael Collins - 2009

1 paper in library cites

[31]An Introduction to Latent Semantic Analysis

T. K. Landauer, P. W. Foltz, D. Laham - 1998

1 paper in library cites

[32]An Introduction to Random Indexing

M. Sahlgren - 2005

1 paper in library cites

[33]Comparison of Independent Component Analysis and Singular Value Decomposition in Word Context Analysis

J. Vayrynen, T. Honkela - 2005

1 paper in library cites

[34]Contextual Relations of Words in Grimm Tales, Analyzed by Self-Organizing Map

T. Honkela, V. Pulkki, T. Kohonen - 1995

1 paper in library cites

[35]Dimensionality Reduction by Random Mapping: Fast Similarity Computation for Clustering

S. Kaski - 1998

1 paper in library cites

[36]Enhancing Unlexicalized Parsing Performance Using a Wide Coverage Lexicon, Fuzzy Tag-Set Mapping, and EM-HMM-based lexical Probabilities

Y. Goldberg, R. Tsarfaty, M. Adler, M. Elhadad - 2009

1 paper in library cites

[37]From Baby Steps to Leapfrog: How "Less Is More" in Unsupervised Dependency Parsing

V. Spitkovsky, H. Alshawi, Dan Jurafsky - 2010

1 paper in library cites

[38]Hierarchical Clustering of Words

A. Ushioda - 1996

1 paper in library cites

[39]Improving Generative Statistical Parsing With Semi-Supervised Word Clustering

M. Candito, B. Crabbe - 2009

1 paper in library cites

[40]Multilingual Dependency Learning: A Huge Feature Engineering Method to Semantic Dependency Parsing

H. Zhao, Weizhu Chen, C. Kit, G. Zhou - 2009

1 paper in library cites

[41]Self-Organizing Maps of Words for Natural Language Processing Applications

T. Honkela - 1997

1 paper in library cites

[42]Self-Organizing Semantic Maps

H. Ritter, T. Kohonen - 1989

1 paper in library cites

[43]Semantic and Associative Priming in High-Dimensional Semantic Space

K. Lund, C. Burgess, R. A. Atchley - 1995

1 paper in library cites

[44]Semi-Supervised Semantic Role Labeling Using the Latent Words Language Model

K. Deschacht, M. F. Moens - 2009

1 paper in library cites

[45]Software Frame-Work for Topic Modelling With Large Corpora

R. Rehurek, P. Sojka - 2010

1 paper in library cites

[46]The Word-Space Model: Using Distributional Analysis to Represent Syn-Tagmatic and Paradigmatic Relations Between Words in High-Dimensional Vector Spaces

M. Sahlgren - 2006

1 paper in library cites

[47]Towards Explicit Semantic Features Using Independent Component Analysis

J. J. Vayrynen, T. Honkela, L. Lindqvist - 2007

1 paper in library cites

[48]Using Latent Semantic Analysis to Improve Access to Textual Information

S. T. Dumais, G. W. Furnas, T. K. Landauer, S. T. Deerwester, R. Harshman - 1988

1 paper in library cites

[49]Vector-Based Semantic Analysis: Representing Word Meanings Based on Random Labels

M. Sahlgren - 2001

1 paper in library cites

[50]Word Cat-Egory Maps Based on Emergent Features Created by ICA

J. J. Vayrynen, T. Honkela - 2004

1 paper in library cites

Cited by

17

papers in your library

Cites

10

papers in your library

Read

on March 20, 2025

This basically introduced the concept of embeddings. Very nice.

Tags

Paper Aliases

No aliases