Papperoni

2014

Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

95

AI summary

This paper introduces a new RNN Encoder-Decoder model for statistical machine translation, using two recurrent neural networks to encode and decode sequences. The model improves translation performance by learning semantically meaningful phrase representations, achieving better results on the English/French translation task of the WMT'14 workshop.

Main Contributions

Introduces a novel neural network architecture called RNN Encoder-Decoder for statistical machine translation.
Proposes a new type of hidden unit with reset and update gates for adaptive control of memory.
Demonstrates improved translation performance on the English/French translation task of the WMT'14 workshop by scoring phrase pairs with the RNN Encoder-Decoder.
Shows that the RNN Encoder-Decoder captures linguistic regularities in phrase pairs and proposes well-formed target phrases.
Visualizes word and phrase representations learned by the model, revealing semantic and syntactic clustering.

Abstract

In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly trained to maximize the conditional probability of a target sequence given a source sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder–Decoder as an additional feature in the existing log-linear model. Qualitatively, we show that the proposed model learns a semantically and syntactically meaningful representation of linguistic phrases.

Citation Graph

Loading graph...

References [32]

Sort:

Filter:

[1]ImageNet Classification With Deep Convolutional Neural Networks

Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - 2012

71 papers in library cite

I'm giving this a 5 just because of the impact, but this is VEEERY derivative of earlier work. Kudos for them for putting it all together, but really there's nothing revolutionary here.

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[4]Deep Sparse Rectifier neural Networks

Xavier Glorot, Antoine Bordes, Yoshua Bengio - 2011

17 papers in library cite

How can you not love Bengio? This paper is everything that Hinton's is not. Really well explained, surprising results, good comparisons... Amazing impact!

[5]A Neural Probabilistic Language Model

Yoshua Bengio, R. Ducharme, Pascal Vincent - 2001

62 papers in library cite

What started it all. Very simple and elegant.

[6]Adadelta: An Adaptive Learning Rate Method

Matthew D. Zeiler - 2012

13 papers in library cite

Cool contribution. Simple and straightforward, and not too mathy. Based on ADAGRAD.

[7]Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

G. Dahl, D. Yu, L. Deng, Alex Acero - 2012

19 papers in library cite

Good paper, very well written and probably the best explanation of RBMs and DBNs I've seen. However, I don't see a lot of impact and seems very derivative from other works.

[8]Maxout Networks

Yoshua Bengio - 2013

17 papers in library cite

A bit hard to understand, but very nice idea.

[9]Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks

Surya Ganguli - 2014

9 papers in library cite

TBH it's been almost 2 months since I read this paper (shame on me for forgetting to add it)... Anyway, as I recall it I liked it, but TBH it's a bit underwhelming because it solved only for linear networks

[10]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[11]Theano: New Features and Speed Improvements

F. Bastien, P. Lamblin, Razvan Pascanu, James Bergstra, I. Goodfellow, A. Bergeron, A. Bouchard, N. Nicolas, Yoshua Bengio - 2012

13 papers in library cite

The paper itself is ok, but it seems like they just wanted to make a paper to say "hey, ours is so much better now after these improvements, we crushed Torch7". A bit tryhard tbh.

[12]How to Construct Deep Recurrent Neural Networks

Razvan Pascanu, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2013

7 papers in library cite

Very interesting despite not being too relevant. Good read and a new way of thinking about RNNs.

[13]Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection

Richard Socher, Eric H. Huang, J. Pennin, C. Manning, A. Ng - 2011

10 papers in library cite

Good paper overall, but seems very simple ideas and complex implementation. Overall not very impactful

[14]Advances in Optimizing Recurrent Networks

Yoshua Bengio, N. B. Lewandowski, Razvan Pascanu - 2013

4 papers in library cite

Amazing paper that summarizes basically everything that was happening up to 2012 and explains why there was a boom in RNN in 2013 onwards.

[15]Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, John Makhoul - 2014

9 papers in library cite

Simple and very nice that it showed that NNs can stand on their own for NMT.

[16]Continuous Space Language Models

Holger Schwenk - 2007

12 papers in library cite

One more paper about speech recog. Nothing special really.

[17]Continuous Space Translation Models for Phrase-Based Statistical Machine Translation

Holger Schwenk - 2012

5 papers in library cite

Nothing special.

[18]Theano: A CPU and GPU Math Expression Compiler

James Bergstra, O. Breuleux, F. Bastien, P. Lamblin, Razvan Pascanu, G. Desjardins, J. Turian, D. W. Farley, Yoshua Bengio - 2010

22 papers in library cite

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[19]Bilingual Word Embeddings for Phrase-Based Machine Translation

W. Zou, Richard Socher, D. Cer, C. Manning - 2013

4 papers in library cite

Bilingual embeddings?

[20]Statistical Phrase-Based Translation

P. Koehn, F. J. Och, D. Marcu - 2003

8 papers in library cite

[21]Supervised Sequence Labelling With Recurrent Neural Networks

Alex Graves - 2012

6 papers in library cite

[22]Decoding With Large-Scale Neural Language Models Improves Translation

Ashish Vaswani, Y. Zhao, V. Fossum, D. Chiang - 2013

5 papers in library cite

[23]Domain Adaptation via Pseudo in-Domain Data Selection

A. Axelrod, X. Fe, Jianfeng Gao - 2011

5 papers in library cite

[24]Continuous Space Translation Models With Neural Networks

L. H. Son, A. Allauzen, F. Yvon - 2012

4 papers in library cite

[25]Joint Language and Translation Modeling With Recurrent Neural Networks

Michael Auli, M. Galley, C. Quirk, Geoffrey Zweig - 2013

3 papers in library cite

[26]Europarl: A Parallel Corpus for Statistical Machine Translation

P. Koehn - 2005

2 papers in library cite

[27]A Phrase-Based, Joint Probability Model for Statistical Machine Translation

D. Marcu, W. Wong - 2002

1 paper in library cites

[28]An Autoencoder Approach to Learning Bilingual Word Representations

S. Chandar, S. Lauly, Hugo Larochelle, M. Khapra, B. Ravindran, V. Raykar, A. Saha - 2014

1 paper in library cites

[29]Barnes-Hut-Sne

Laurens Van Der Maaten - 2013

1 paper in library cites

[30]Continuous Space Language Models for the iwslt 2006 Task

Holger Schwenk, M. R. C. Jussa, J. A. R. Fonollosa - 2006

1 paper in library cites

[31]Intelligent Selection of Language Model Training Data

R. C. Moore, W. Lewis - 2010

1 paper in library cites

[32]Learning Semantic Representations for the Phrase Translation Model

Jianfeng Gao, X. He, W. Yih, L. Deng - 2013

1 paper in library cites

Cited by

38

papers in your library

Cites

19

papers in your library

Read

on June 7, 2025

Introduces RNN encoder-decoder. I love it :)

Tags

Paper Aliases

No aliases