Papperoni

2017

Convolutional Sequence to Sequence Learning

J. Gehring, Michael Auli, D. Grangier, D. Yarats, Yann Dauphin

Open PDF Google Scholar

citations

Cite Score

72

AI summary

This paper introduces a sequence-to-sequence model based entirely on convolutional neural networks (CNNs) with gated linear units and attention modules. The model outperforms deep LSTM models on WMT'14 English-German and English-French translation tasks, achieving faster speeds on both GPU and CPU.

Main Contributions

Introduces a fully convolutional architecture for sequence-to-sequence learning, enabling parallelization and easier optimization compared to recurrent models.
Employs gated linear units to improve gradient propagation and equips each decoder layer with a separate attention module.
Achieves state-of-the-art results on WMT'16 English-Romanian translation, outperforming the winning entry by 1.9 BLEU.
Outperforms deep LSTM models on WMT'14 English-German and English-French translation tasks, with significant improvements in BLEU scores.
Demonstrates faster translation speeds compared to LSTM-based models on both GPU and CPU hardware.

Abstract

The prevalent approach to sequence to sequence learning maps an input sequence to a variable length output sequence via recurrent neural networks. We introduce an architecture based entirely on convolutional neural networks. Compared to recurrent models, computations over all elements can be fully parallelized during training to better exploit the GPU hardware and optimization is easier since the number of non-linearities is fixed and independent of the input length. Our use of gated linear units eases gradient propagation and we equip each decoder layer with a separate attention module. We outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT' 14 English-French translation at an order of magnitude faster speed, both on GPU and CPU.

Citation Graph

Loading graph...

References [48]

Sort:

Filter:

[1]Deep Residual Learning for Image Recognition

K. He, X. Zhang, S. Ren, Jian Sun - 2016

20 papers in library cite

This is simply amazing. Very very simple idea, totally revolutionary. No maths, just "it works!". Amazing.

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

S. Ioffe, Christian Szegedy - 2015

18 papers in library cite

Very good paper! Similar feel as ResNets: simple idea, elegant. Not too mathy

[4]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[5]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[6]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[7]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[8]Understanding the Difficulty of Training Deep Feedforward Neural Networks

Yoshua Bengio - 2010

20 papers in library cite

Nice but underwhelming results (they still underperform vs. pretraining). I also didn't really like the way it's written. It's not bad, it's just a bit clunky. Worth the read though.

[9]Delving Deep Into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

K. He, X. Zhang, S. Ren, Jian Sun - 2015

10 papers in library cite

I think the PRELU idea didn't catch on, but the initialization is very nice! Good read.

[10]ROUGE: A Package for Automatic Evaluation of Summaries

Chin Yew Lin - 2004

9 papers in library cite

Wow, very poorly written and badly explained. This is the exact opposite of BLEU

[11]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[12]Layer Normalization

Jimmy Lei Ba, R. Kiros, Geoffrey E. Hinton - 2016

14 papers in library cite

Very nice! At first I had a little bit of prejudice because it seemed way too mathy, but actually the math is easy to follow and the results are very nice.

[13]Effective Approaches to Attention-Based Neural Machine Translation

T. Luong, H. Pham, Christopher D. Manning - 2015

15 papers in library cite

Good paper, but very derivative. Attention methods start getting very complicated... I understand why Transformers took over TBH

[14]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[15]Google's Neural Machine Translation System: Bridging the Gap Between Human and Machine Translation

Yonghui Wu, M. Schuster, Ziru Chen, Quoc V. Le, M. Norouzi, W. Macherey, M. Krikun, Yue Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. J. Johnson, Xiaodong Liu, Lukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, Wenyi Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, Oriol Vinyals, G. S. Corrado, M. Hughes, Jeffrey Dean - 2016

15 papers in library cite

It's a very good paper but TBH doesn't bring anything new other than joining a bunch of existing stuff. I think it ended up being foundational because it's Google and several people used it as a base for future research. Good contribution then :)

[16]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[17]On the Importance of Initialization and Momentum in Deep Learning

Ilya Sutskever, James Martens, G. Dahl, Geoffrey Hinton - 2013

13 papers in library cite

They give very good context and it's easy to understand that they are doing this as a counterpoint to HF. Surprising results as well. I just think it was made obsolete by relu

[18]Phoneme Recognition Using Time-Delay Neural Networks

A. H. Waibel, T. Hanazawa, Geoffrey Hinton, K. Shikano, K. Lang - 1989

13 papers in library cite

I didn't really like this paper, although it's a good early example of using backprop. I think it can be thought as a 1D convolution.

[19]A Neural Attention Model for Abstractive Sentence Summarization

Alexander M. Rush, S. Chopra, Jason Weston - 2015

13 papers in library cite

TBH the paper is a bit boring and nothing new after reading a bunch of more modern techniques. I feel that they could have done a better job considering that seq2seq existed at the time. Either way, points for being the first to propose summarization with NNs.

[20]Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Noam Shazeer, Azalia Mirhoseini, K. Maziarz, A. Davis, Quoc Le, Geoffrey Hinton, Jeffrey Dean - 2017

9 papers in library cite

It's nice, but there's an important section in the middle about batch sizes that I don't quite understand. Not sure if I am missing some background knowledge or if they explain it poorly, and seems foundational to their main method... Either way, I did understand the methodology of the paper, and they have nice results :)

[21]End-to-End Memory Networks

S. Sukhbaatar, A. Szlam, Jason Weston, Rob Fergus - 2015

18 papers in library cite

This was so surprising! This is very similar to transformers and RAG. Who knew?!

[22]Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond

R. Nallapati, B. Zhou, C. N. D. Santos, C. G. Gulcehre, Bing Xiang - 2016

10 papers in library cite

Cool that they use pointer switching, and they introduce the CNN/daily mail dataset for summarization, which is a nice insight.

[23]Torch7: A Matlab-Like Environment for Machine Learning

Clement Farabet - 2011

5 papers in library cite

Very nice paper! I am impressed that they used Lua, and the API sounds very simple. Plus, it sounds like they didn't even try that hard to get good performance (vs. Theano which was very tryhard)

[24]Japanese and Korean Voice Search

M. Schuster, Kaisuke Nakajima - 2012

3 papers in library cite

This article sucks. Very poorly written and the only nice thing (wordpiece) is very poorly explained

[25]Language Modeling With Gated Convolutional Networks

Yann N. Dauphin, A. Fan, Michael Auli, D. Grangier - 2016

8 papers in library cite

[26]Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

T. Salimans, D. A. Kingma, D. P. Diederik - 2016

4 papers in library cite

Cited by a lot of people

[27]HyperNetworks

D. Ha, Andrew Dai, Quoc V. Le - 2016

3 papers in library cite

Using a network to generate weights for other network

[28]Key-Value Memory Networks for Directly Reading Documents

A. Miller, Adam Fisch, J. Dodge, A. Karimi, Antoine Bordes, Jason Weston - 2016

1 paper in library cites

Key-Value memory networks?

[29]Neural Machine Translation in Linear Time

N. Kalchbrenner, L. Espeholt, K. Simonyan, A. V. D. Oord, Alex Graves, Koray Kavukcuoglu - 2016

5 papers in library cite

Bytenet - Also "linear time" caught my attention

[30]Quasi-Recurrent Neural Networks

J. Bradbury, S. Merity, Caiming Xiong, Richard Socher - 2016

1 paper in library cites

Title caught my attention, but only one internal citation...

[31]Pixel Recurrent Neural Networks

A. V. D. Oord, N. Kalchbrenner, Koray Kavukcuoglu - 2016

3 papers in library cite

Seems like they introduce input masking

[32]Cutting-Off Redundant Repeating Generations for Neural Abstractive Summarization

J. Suzuki, M. Nagata - 2017

2 papers in library cite

How did they manage to reduce repeated info?

[33]Deep Recurrent Models With Fast-Forward Connections for Neural Machine Translation

Jingren Zhou, Yue Cao, Xinpeng Wang, P. L. Li, Weixin Xu - 2016

5 papers in library cite

[34]Edinburgh neural Machine Translation Systems for WMT 16

R. Sennrich, B. Haddow, Alexandra Birch - 2016

5 papers in library cite

[35]English Gigaword

R. Parker, D. Graff, J. Kong, K. Chen, K. Maeda - 2011

5 papers in library cite

[36]A Simple, Fast, and Effective Reparameterization of IBM Model 2

C. Dyer, V. Chahuneau, Noah A. Smith - 2013

4 papers in library cite

[37]Attention-Based Models for Speech Recognition

J. Chorowski, D. Bahdanau, D. Serdyuk, Kyunghyun Cho, Yoshua Bengio - 2015

3 papers in library cite

[38]Convolutional Networks for Images, Speech, and Time-Series

Yann Lecun, Yoshua Bengio - 1995

3 papers in library cite

[39]Encoding Source Language With Convolutional Neural Network for Machine Translation

Fanqing Meng, Z. L. Lu, Mingliang Wang, H. Li, W. Jiang, Qian Liu - 2015

3 papers in library cite

[40]Montreal Neural Machine Translation Systems for WMT'15

S. Jean, O. Firat, Kyunghyun Cho, R. Memisevic, Yoshua Bengio - 2015

3 papers in library cite

[41]A Convolutional Encoder Model for Neural Machine Translation

J. Gehring, Michael Auli, D. Grangier, Yann N. Dauphin - 2016

2 papers in library cite

[42]Duc in Context

P. Over, H. Dang, D. Harman - 2007

2 papers in library cite

[43]Conditional Image Generation With pixelcnn Decoders

A. Oord, N. Kalchbrenner, Oriol Vinyals, L. Espeholt, Alex Graves, Koray Kavukcuoglu - 2016

1 paper in library cites

[44]Findings of the 2016 Conference on Machine Translation

O. Bojar, R. Chatterjee, C. Federmann, C. Graham, Y. Yvette, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, V. Logacheva, C. Monz, M. Negri, A. Neveol, M. Neves, M. Popel, M. Post, R. Rubino, C. Scarton, L. Specia, M. Turchi, K. Verspoor, M. Zampieri - 2016

1 paper in library cites

[45]Neural Headline Generation With Sentence-Wise Optimization

S. Shen, Y. Zhao, Ze Liu, Maosong Sun - 2016

1 paper in library cites

[46]Neural Machine Translation With Recurrent Attention Modeling

Zhilin Yang, Z. Hu, Y. Deng, C. Dyer, A. Smola - 2016

1 paper in library cites

[47]Vocabulary Manipulation for Neural Machine Translation

H. Mi, Zhengtao Wang, A. Ittycheriah - 2016

1 paper in library cites

[48]Vocabulary Selection Strategies for Neural Machine Translation

G. L'hostis, D. Grangier, Michael Auli - 2016

1 paper in library cites

Cited by

3

papers in your library

Cites

32

papers in your library

Read

on August 23, 2025

It's a good paper. I liked that it is very similar to the transformer methodology (but they don't use convolutions there). I need to go back and read the transformers paper to make parallels.

Tags

Paper Aliases

No aliases