Papperoni

2011

Natural Language Processing (Almost) From Scratch

Ronan Collobert, Jason Weston, Leon Bottou, M. Karlen, Koray Kavukcuoglu, P. P. Kuksa

Open PDF Google Scholar

citations

Cite Score

85

AI summary

This paper introduces a unified neural network architecture for NLP tasks, learning internal representations from large, unlabeled data. It achieves good performance in part-of-speech tagging, chunking, named entity recognition, and semantic role labeling, demonstrating the effectiveness of its "almost from scratch" approach.

Main Contributions

Proposes a unified neural network architecture applicable to various NLP tasks, minimizing task-specific engineering.
Learns internal representations from large, mostly unlabeled data, avoiding reliance on hand-crafted features.
Demonstrates versatility across part-of-speech tagging, chunking, named entity recognition, and semantic role labeling.
Reaches good performance levels by transferring intermediate representations from large unlabeled datasets.
Builds a freely available tagging system with good performance and minimal computational requirements.

Abstract

We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

Citation Graph

Loading graph...

References [91]

Sort:

Filter:

[1]Gradient-Based Learning Applied to Document Recognition

Yann Lecun, Leon Bottou, Yoshua Bengio, Patrick Haffner - 1998

62 papers in library cite

I absolutely hated this paper. Has ~50 pages but seems like 200 pages. Takes too long to explain some things that really is just repeating itself. Also doesn't seem to add too much on top of LeNet-5. Also, focuses a lot on GTNs, which really didn't stick.

[2]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[3]A Fast Learning Algorithm for Deep Belief Nets

Geoffrey E. Hinton, S. Osindero, Y. Teh - 2006

43 papers in library cite

The paper does not explain anything. It just throws the idea and a bunch of math, but doesn't really care to explain the concepts.

[4]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[5]Multitask Learning

Rich Caruana - 1997

13 papers in library cite

I expected waaaaaay more from this paper. The idea is sooooo simple and the results are underwhelming. Also, 30 pages for something that could be said in 10. The writing style is a bit boring. TBH it seems like it's just a re-writing of Caruana's PhD thesis.

[6]Efficient Backprop

Yann Lecun, Leon Bottou, G. B. Orr, Klaus Robert Muller - 1998

20 papers in library cite

The first half is very very good. The remainder is very boring.

[7]Greedy Layer-Wise Training of Deep Networks

Yoshua Bengio, P. Lamblin, D. Popovici, Hugo Larochelle - 2006

33 papers in library cite

Bengio is perfect. This is everything that Hinton's paper hoped to be. Very well explained, and also tying back to real use cases (not just "hey, the math works and it reduced the score")

[8]Curriculum Learning

Yoshua Bengio, J. Louradour, Ronan Collobert, Jason Weston - 2009

6 papers in library cite

Very nice paper that introduces curriculum learning. Possibly not too relevant, but good nonetheless.

[9]Phoneme Recognition Using Time-Delay Neural Networks

A. H. Waibel, T. Hanazawa, Geoffrey Hinton, K. Shikano, K. Lang - 1989

13 papers in library cite

I didn't really like this paper, although it's a good early example of using backprop. I think it can be thought as a 1D convolution.

[10]Word Representations: A Simple and General Method for Semi-Supervised Learning

J. Turian, L. Ratinov, Yoshua Bengio - 2010

17 papers in library cite

This basically introduced the concept of embeddings. Very nice.

[11]Probabilistic Interpretation of Feedforward Classification Network Outputs, With Relationship to Statistical Pattern Recognition

John S. Bridle - 1990

12 papers in library cite

Explains using softmax as output. Nice read, but I am not sure if too relevant.

[12]A Framework for Learning Predictive Structures From Multiple Tasks and Unlabeled Data

Rie Kubota Ando, Tong Zhang - 2005

10 papers in library cite

Very nice and clever way of solving the problem of semi-supervised learning, and makes a lot of sense. I give them more credit for formalizing the concept. The methodology is a bit boring.

[13]Deep Learning via Semi-Supervised Embedding

Jason Weston, F. Ratle, Ronan Collobert - 2008

10 papers in library cite

It's a good paper and nice idea, but seems overly complicated and I don't think it's very used... (PS: this was republished in 2012)

[14]Three New Graphical Models for Statistical Language Modelling

A. Mnih, Geoffrey Hinton - 2007

12 papers in library cite

I don't know why this is so impactful. I didn't like it and I think this was overly complex.

[15]Connectionist Language Modeling for Large Vocabulary Continuous Speech Recognition

Holger Schwenk, Jean Luc Gauvain - 2002

14 papers in library cite

Only real relevance is being early. Otherwise not much to see.

[16]A Learning Scheme for Asymmetric Threshold Networks

Yann Lecun - 1985

5 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[17]Class-Based N-Gram Models of Natural Language

P. F. Brown, P. V. Desouza, R. L. Mercer, Vincent J. Della Pietra, J. C. Lai - 1992

12 papers in library cite

[18]Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

J. Lafferty, Andrew Mccallum, F. C. Pereira - 2001

6 papers in library cite

[19]Semi-Supervised Learning

O. Chapelle, B. Scholkopf, A. Zien - 2006

5 papers in library cite

[20]Transductive Inference for Text Classification Using Support Vector Machines

T. Joachims - 1999

5 papers in library cite

[21]A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition

L. R. Rabiner - 1989

4 papers in library cite

[22]Effective Self-Training for Parsing

D. Mcclosky, E. Charniak, M. J. Johnson - 2006

4 papers in library cite

[23]Online Algorithms and Stochastic Approximations

Leon Bottou - 1998

4 papers in library cite

[24]Semi-Supervised Sequential Labeling and Segmentation Using Gigaword Scale Unlabeled Data

J. Suzuki, H. Isozaki - 2008

4 papers in library cite

[25]Bayesian Learning for Neural Networks

R. Neal - 1995

3 papers in library cite

[26]Dependency Networks for Inference, Collaborative Filtering, and Data Visualization

D. Heckerman, D. M. Chickering, C. Meek, R. Rounthwaite, C. Kadie - 2000

3 papers in library cite

[27]Design Challenges and Misconceptions in Named Entity Recognition

L. A. Ratinov, Dan Roth - 2009

3 papers in library cite

[28]Named Entity Recognition Through classifier Combination

R. Florian, A. Ittycheriah, H. Jing, Tong Zhang - 2003

3 papers in library cite

[29]Shallow Semantic Parsing Using Support Vector Machines

S. Pradhan, W. Ward, K. Hacioglu, J. Martin, Dan Jurafsky - 2004

3 papers in library cite

[30]Symbolic-Neural Systems and the Use of Hints for Developing Complex Systems

S. C. Suddarth, A. D. C. Holden - 1991

3 papers in library cite

[31]The Proposition Bank: An Annotated Corpus of Semantic Roles

M. Palmer, P. Kingsbury, D. Gildea - 2005

3 papers in library cite

[32]A Discriminative Language Model With Pseudo-Negative Samples

D. Okanohara, J. Tsujii - 2007

2 papers in library cite

[33]A Framework for the Cooperation of Learning Algorithms

Leon Bottou, P. Gallinari - 1991

2 papers in library cite

[34]A Maximum-Entropy-Inspired Parser

E. Charniak - 2000

2 papers in library cite

[35]A Novel Use of Statistical Parsing to Extract Information From Text

S. Miller, H. Fox, L. Ramshaw, R. Weischedel - 2000

2 papers in library cite

[36]Composition of Conditional Random Fields for Transfer Learning

Charles Sutton, Andrew Mccallum - 2005

2 papers in library cite

[37]Continuous Speech Recognition by Statistical Methods

Frederick Jelinek - 1976

2 papers in library cite

[38]Contrastive estimation: Training Log-Linear Models on Unlabeled Data

Noah A. Smith, J. Eisner - 2005

2 papers in library cite

[39]Deep Learning for Efficient Discriminative Parsing

Ronan Collobert - 2011

2 papers in library cite

[40]Distributional Representations for Handling Sparsity in Supervised Sequence-Labeling

F. Huang, A. Yates - 2009

2 papers in library cite

[41]Dynamic Conditional Random Fields: Factorized Probabilistic Models for Labeling and Segmenting Sequence Data

Charles Sutton, Andrew Mccallum, K. Rohanimanesh - 2007

2 papers in library cite

[42]Feature-Rich Part-of-Speech Tagging With a Cyclic Dependency Network

Kristina Toutanova, Dan Klein, Christopher D. Manning, Yoram Singer - 2003

2 papers in library cite

[43]Global Training of Document Processing Systems Using Graph Transformer Networks

Leon Bottou, Yoshua Bengio, Yann Lecun - 1997

2 papers in library cite

[44]Head-Driven Statistical Models for Natural Language Parsing

Michael Collins - 1999

2 papers in library cite

[45]Joint Parsing and Semantic Role Labeling

Charles Sutton, Andrew Mccallum - 2005

2 papers in library cite

[46]Learning Quickly When Irrelevant Attributes Abound: A New Linear-Threshold Algorithm

N. Littlestone - 1988

2 papers in library cite

[47]Learning Sets of Filters Using Back-Propagation

D. C. Plaut, Geoffrey E. Hinton - 1987

2 papers in library cite

[48]Name Tagging With Word Clusters and Discriminative Training

S. Miller, J. Guinness, A. Zamanian - 2004

2 papers in library cite

[49]Named Entity Recognition With a Maximum Entropy Approach

H. L. Chieu - 2003

2 papers in library cite

[50]Phrase Clustering for Discriminative Learning

D. Lin, Xiaobao Wu - 2009

2 papers in library cite

[51]Prediction and Entropy of Printed english

C. E. Shannon - 1951

2 papers in library cite

[52]Rcv1: A New Benchmark Collection for Text Categorization Research

D. D. Lewis, Yining Yang, T. G. Rose, F. Li - 2004

2 papers in library cite

[53]Robust Parsing of the Proposition Bank

G. Musillo, P. Merlo - 2006

2 papers in library cite

[54]Semi-Supervised Learning for Natural Language

Percy Liang - 2005

2 papers in library cite

[55]Shallow Parsing With Conditional Random Fields

F. Sha, Fernando Pereira - 2003

2 papers in library cite

[56]Simple Semi-Supervised Dependency Parsing

T. Koo, X. Carreras, Michael Collins - 2008

2 papers in library cite

[57]Stochastic Gradient Learning in Neural Networks

Leon Bottou - 1991

2 papers in library cite

[58]The BellKor Solution to the Netflix Prize

R. M. Bell, Y. Koren, C. Volinsky - 2007

2 papers in library cite

[59]The Necessity of Parsing for Predicate Argument Recognition

D. Gildea, M. Palmer - 2001

2 papers in library cite

[60]Three Models for the Description of Language

N. Chomsky - 1956

2 papers in library cite

[61]Transductive Learning for Statistical Machine Translation

N. Ueffing, G. Haffari, A. Sarkar - 2007

2 papers in library cite

[62]Using Corpus Statistics on Entities to Improve Semi-Supervised Relation Extraction From the Web

B. Rosenfeld, R. Feldman - 2007

2 papers in library cite

[63]A Convergent Gambling Estimate of the Entropy of english

T. Cover, R. King - 1978

1 paper in library cites

[64]A Joint Model for Semantic Role Labeling

A. Haghighi, Kristina Toutanova, Christopher D. Manning - 2005

1 paper in library cites

[65]A Maximum Entropy Model for Part-of-Speech Tagging

A. Ratnaparkhi - 1996

1 paper in library cites

[66]An Algorithm for Suffix Stripping

M. F. Porter - 1980

1 paper in library cites

[67]An Estimate of an Upper Bound for the Entropy of english

P. F. Brown, Vincent J. Della Pietra, R. L. Mercer, S. A. D. Pietra, J. C. Lai - 1992

1 paper in library cites

[68]Automatic Labeling of Semantic Roles

D. Gildea, Dan Jurafsky - 2002

1 paper in library cites

[69]Chunking With Support Vector Machines

T. Kudo, Y. Matsumoto - 2001

1 paper in library cites

[70]Comparing and Combining Finite-State and Context-Free Parsers

K. Hollingshead, S. Fisher, B. Roark - 2005

1 paper in library cites

[71]Distributional Part-of-Speech Tagging

Hinrich Schutze - 1995

1 paper in library cites

[72]Early Results for Named Entity Recognition With Conditional Random Fields, Feature Induction and Web-Enhanced Lexicons

Andrew Mccallum, Wentao Li - 2003

1 paper in library cites

[73]Flexible Text Segmentation With Structured Multilabel Classification

R. Mcdonald, K. Crammer, Fernando Pereira - 2005

1 paper in library cites

[74]Generalized Inference With Multiple Semantic Role Labeling Systems (Shared Task Paper)

P. Koomen, V. Punyakanok, Dan Roth, W. Yih - 2005

1 paper in library cites

[75]Guided Learning for Bidirectional Sequence Classification

L. Shen, G. Satta, A. K. Joshi - 2007

1 paper in library cites

[76]Large Scale Machine Learning

Ronan Collobert - 2004

1 paper in library cites

[77]Learning to Order Things

W. W. Cohen, R. E. Schapire, Yoram Singer - 1998

1 paper in library cites

[78]Learning to Rank With Nonsmooth Cost Functions

C. J. C. Burges, R. Ragno, Quoc V. Le - 2007

1 paper in library cites

[79]Mathematical Structures of Language

Z. S. Harris - 1968

1 paper in library cites

[80]Modeling Latent-Dynamic in Shallow Parsing: A Latent Conditional Model With Improved Inference

X. Sun, Louis Philippe Morency, D. Okanohara, J. Tsujii - 2008

1 paper in library cites

[81]Natural Language Grammar Induction Using a Constituent-Context Model

Dan Klein, Christopher D. Manning - 2002

1 paper in library cites

[82]Probabilistic Reasoning in Intelligent Systems

J. Pearl - 1988

1 paper in library cites

[83]Ranking the Best Instances

S. Clemencon, N. Vayatis - 2007

1 paper in library cites

[84]Semantic Role Chunking Combining Complementary Syntactic Views

S. Pradhan, K. Hacioglu, W. Ward, J. H. Martin, Dan Jurafsky - 2005

1 paper in library cites

[85]Semantic Role Labelling With Tree Conditional Random Fields

T. Cohn, Phil Blunsom - 2005

1 paper in library cites

[86]Structure Compilation: Trading Structure for Features

Percy Liang, H. Daume, Dan Klein - 2008

1 paper in library cites

[87]SVMTool: A General POS Tagger Generator Based on Support Vector Machines

J. Gimenez, L. Marquez - 2004

1 paper in library cites

[88]The Entropy of english Using PPM-Based Models

W. J. Teahan, J. G. Cleary - 1996

1 paper in library cites

[89]The Necessity of Syntactic Parsing for Semantic Role Labeling

V. Punyakanok, Dan Roth, W. Yih - 2005

1 paper in library cites

[90]Use of Support Vector Learning for Chunk Identification

T. Kudoh, Y. Matsumoto - 2000

1 paper in library cites

[91]Voting Between Multiple Data Representations for Text Chunking

H. Shen, A. Sarkar - 2005

1 paper in library cites

Cited by

23

papers in your library

Cites

16

papers in your library

Read

on October 15, 2025

It's one of those huge papers that gets very tiring by the end. However, it's a very nice contribution. I am biased towards not liking it because it's basically old style NLP using NNs, which to me is a bit meh. However, I think this sets very important foundations for pretraining, embeddings, and proving that NNs rock.

Tags

Paper Aliases

No aliases