Papperoni

2015

Grammar as a Foreign Language

Geoffrey Hinton

Open PDF Google Scholar

citations

Cite Score

39

AI summary

This paper introduces an attention-enhanced sequence-to-sequence model for syntactic constituency parsing, achieving state-of-the-art results on the WSJ dataset when trained on a large synthetic corpus. The model is also data-efficient and fast, processing over a hundred sentences per second on a CPU.

Main Contributions

Introduces an attention-enhanced sequence-to-sequence model for syntactic constituency parsing.
Achieves state-of-the-art results on the WSJ dataset when trained on a large synthetic corpus annotated using existing parsers.
Demonstrates high data efficiency compared to sequence-to-sequence models without attention.
Achieves fast parsing speed, processing over a hundred sentences per second on a CPU.
Shows that domain independent models with excellent learning algorithms can match and even outperform domain specific models.

Abstract

Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades. As a result, the most accurate parsers are domain specific, complex, and inefficient. In this paper we show that the domain agnostic attention-enhanced sequence-to-sequence model achieves state-of-the-art results on the most widely used syntactic constituency parsing dataset, when trained on a large synthetic corpus that was annotated using existing parsers. It also matches the performance of standard parsers when trained only on a small human-annotated dataset, which shows that this model is highly data-efficient, in contrast to sequence-to-sequence models without the attention mechanism. Our parser is also fast, processing over a hundred sentences per second with an unoptimized CPU implementation.

Citation Graph

Loading graph...

References [32]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

26 papers in library cite

Expanded wor2vec. Very nice overall.

[3]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[4]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[5]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[6]Show and Tell: A Neural Image Caption Generator

Dumitru Erhan - 2015

11 papers in library cite

It's nice and they beat a ton of SotA. However, I read the one that uses attention first so this is a bit less surprising.

[7]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[8]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[9]Parsing Natural Scenes and Natural Language With Recursive Neural Networks

Richard Socher, C. C. Lin, C. Manning, Andrew Y. Ng - 2011

10 papers in library cite

Good idea and nice results, but not my thing right now.

[10]On Using Very Large Target Vocabulary for Neural Machine Translation

Yoshua Bengio - 2014

12 papers in library cite

It's nice, but it starts getting a bit into the realm of "yeah, that seems like a minor improvement". It's nice that they use the importance sampling stuff from the previous paper though - I thought it had completely vanished :)

[11]Addressing the Rare Word Problem in Neural Machine Translation

T. Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, Wojciech Zaremba - 2014

14 papers in library cite

The method was very poorly explained. It was also worse than a paper released sooner, and more complicated. Overall not that good.

[12]Accurate Unlexicalized Parsing

Dan Klein, Christopher D. Manning - 2003

7 papers in library cite

[13]Effective Self-Training for Parsing

D. Mcclosky, E. Charniak, M. J. Johnson - 2006

4 papers in library cite

[14]Learning Accurate, Compact, and Interpretable Tree Annotation

Slav Petrov, L. Barrett, R. Thibaux, Dan Klein - 2006

4 papers in library cite

[15]Constituent Parsing With Incremental Sigmoid Belief Networks

I. Titov, J. Henderson - 2007

2 papers in library cite

[16]Deep Learning for Efficient Discriminative Parsing

Ronan Collobert - 2011

2 papers in library cite

[17]End-to-End Continuous Speech Recognition Using Attention-Based Recurrent Nn: First Results

J. Chorowski, D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

2 papers in library cite

[18]Fast and Accurate Shift-Reduce Constituent Parsing

M. Zhu, Y. Z. Zhang, Weizhu Chen, Mingchuan Zhang, Jiacheng Zhu - 2013

2 papers in library cite

[19]Self-Training PCFG Grammars With Latent Annotations Across Languages

Zhongqiang Huang, M. Harper - 2009

2 papers in library cite

[20]A Linear Observed Time Statistical Parser Based on Maximum Entropy Models

A. Ratnaparkhi - 1997

1 paper in library cites

[21]A Neural Network for Learning How to Parse Tree Adjoining Grammar

Zoubin Ghahramani - 1990

1 paper in library cites

[22]Ambiguity-Aware Ensemble Training for Semi-Supervised Dependency Parsing

Zhiyuan Li, Mingchuan Zhang, Weizhu Chen - 2014

1 paper in library cites

[23]Discriminative Training of a Neural Network Statistical Parser

J. Henderson - 2004

1 paper in library cites

Missing year

[24]Incremental Parsing With the perceptron algorithm

Michael Collins, B. Roark

1 paper in library cites

[25]Inducing History Representations for Broad Coverage Statistical Parsing

J. Henderson - 2003

1 paper in library cites

[26]Ontonotes: The 90% Solution

Eduard Hovy, M. Marcus, M. Palmer, L. Ramshaw, R. Weischedel - 2006

1 paper in library cites

[27]Overview of the 2012 Shared Task on Parsing the Web

Slav Petrov, R. Mcdonald - 2012

1 paper in library cites

[28]Products of Random Latent Variable Grammars

Slav Petrov - 2010

1 paper in library cites

[29]Questionbank: Creating a Corpus of Parse-Annotated Questions

J. Judge, A. Cahill, J. V. Genabith - 2006

1 paper in library cites

[30]Self-Training With Products of Latent Variable Grammars

Zhongqiang Huang, M. Harper, Slav Petrov - 2010

1 paper in library cites

[31]Sparser, Better, Faster GPU Parsing

D. Hall, T. B. Kirkpatrick, J. Canny, Dan Klein - 2014

1 paper in library cites

[32]Three Generative, Lexicalised Models for Statistical Parsing

Michael Collins - 1997

1 paper in library cites

Cited by

9

papers in your library

Cites

11

papers in your library

Read

on August 4, 2025

It's a nice paper showing that attention can be used for parsing. However, parsing is boring and is very derivative. Good paper nonetheless.

Tags

Paper Aliases

No aliases