Papperoni

2017

Unsupervised Pretraining for Sequence to Sequence Learning

P. Ramachandran, P. J. Liu, Quoc V. Le

Open PDF Google Scholar

citations

Cite Score

17

AI summary

This paper introduces a simple and effective technique for using unsupervised pretraining to improve seq2seq models, initializing encoder and decoder networks with pretrained language model weights and fine-tuning with labeled data, achieving state-of-the-art results on WMT English→German.

Main Contributions

Proposes a simple and effective technique for using unsupervised pretraining to improve seq2seq models.
Initializes both encoder and decoder networks with pretrained weights of two language models.
Jointly trains the seq2seq objective with the language modeling objectives to prevent overfitting.
Achieves state-of-the-art results on the WMT English→German task.
Finds that pretraining improves the generalization of seq2seq models.

Abstract

This work presents a general unsupervised learning method to improve the accuracy of sequence to sequence (seq2seq) models. In our method, the weights of the encoder and decoder of a seq2seq model are initialized with the pretrained weights of two language models and then fine-tuned with labeled data. We apply this method to challenging benchmarks in machine translation and abstractive summarization and find that it significantly improves the subsequent supervised models. Our main result is that pretraining improves the generalization of seq2seq models. We achieve state-of-the-art results on the WMT English→German task, surpassing a range of methods using both phrase-based machine translation and neural machine translation. Our method achieves a significant improvement of 1.3 BLEU from the previous best models on both WMT'14 and WMT'15 English German. We also conduct human evaluations on abstractive summarization and find that our method outperforms a purely supervised learning baseline in a statistically significant manner.

Citation Graph

Loading graph...

References [39]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Distributed Representations of Words and Phrases and Their Compositionality

Tomas Mikolov, Ilya Sutskever, K. Chen, G. S. Corrado, Jeffrey Dean - 2013

32 papers in library cite

Introduced word2vec. Game changer.

[3]GloVe: Global Vectors for Word Representation

Jeffrey Pennington, Richard Socher, Christopher D. Manning - 2014

31 papers in library cite

Not a bad paper, I just don't like the motivation and I think the methodology is poorly explained and hard to follow. I can't deny the good results though...

[4]Neural Machine Translation by Jointly Learning to Align and Translate

D. Bahdanau, Kyunghyun Cho, Yoshua Bengio - 2014

59 papers in library cite

Introduces the attention mechanism - amazing overall

[5]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[6]BLUE: A Method for Automatic Evaluation of Machine Translation

K. Papineni, S. Roukos, T. Ward, Wei Jing Zhu - 2002

19 papers in library cite

Very cool idea. Simple yet very impactful!

[7]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[8]ROUGE: A Package for Automatic Evaluation of Summaries

Chin Yew Lin - 2004

9 papers in library cite

Wow, very poorly written and badly explained. This is the exact opposite of BLEU

[9]Neural Machine Translation of Rare Words with Subword Units

R. Sennrich, B. Haddow, Alexandra Birch - 2016

22 papers in library cite

Very good! Simple, explains quite a lot and good results. Forms the basis for a lot of stuff now!

[10]On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio - 2013

21 papers in library cite

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

[11]Teaching Machines to Read and Comprehend

K. M. Hermann, T. Kocisky, Edward Grefenstette, L. Espeholt, W. Kay, M. Suleyman, Phil Blunsom - 2015

31 papers in library cite

Nice way of converting unsupervised data to train for Q&A - and nice visualizations as well :) But I think their main contribution is the dataset. Maybe with the dataset they "unlocked" summarization?

[12]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[13]Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition

G. Dahl, D. Yu, L. Deng, Alex Acero - 2012

19 papers in library cite

Good paper, very well written and probably the best explanation of RBMs and DBNs I've seen. However, I don't see a lot of impact and seems very derivative from other works.

[14]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[15]Semi-Supervised Sequence Learning

A. M. Dai, Quoc V. Le - 2015

27 papers in library cite

Very good paper that was probably the first to introduce pre-training in NLP!

[16]Exploring the Limits of Language Modeling

R. Jozefowicz, Oriol Vinyals, M. Schuster, Noam Shazeer, Yonghui Wu - 2016

20 papers in library cite

It's funny because at first I did not like it, but then it clicked and I really liked it - they are trying to come around the large dictionary and the rare word problem. In the end it's SotA, but I think it's too convoluted and was replaced by Transformers.

[17]Multi-Task Sequence to Sequence Learning

M. T. Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, Lukasz Kaiser - 2015

4 papers in library cite

Very nice paper, but all of the things that I read about S2S sound very derivative at this point - nonetheless it's nice to see MTL again - seems a bit uncharted territory at this point, despite being very promising

[18]Learning to Generate Reviews and Discovering Sentiment

Alec Radford, R. Jozefowicz, Ilya Sutskever - 2017

8 papers in library cite

Very impressive results. We can see that they were onto something (later became GPT2)

[19]Improving Neural Machine Translation Models With Monolingual Data

R. Sennrich, B. Haddow, Alexandra Birch - 2016

4 papers in library cite

The most cited for multilingual/monolingual NMT (out of other references)

[20]Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling

H. Sak, A. W. Senior, F. Beaufays - 2014

5 papers in library cite

Only 5 pages

[21]Sequence-to-Sequence RNNS for Text Summarization

R. Nallapati, Bing Xiang, B. Zhou - 2016

2 papers in library cite

Very well cited, but maybe the same as the other ones

[22]An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks

Ian J. Goodfellow, M. Mirza, D. Xiao, Aaron Courville, Yoshua Bengio - 2014

2 papers in library cite

Meta examination of catastrophic fogetting - nice!

[23]Listen, Attend and Spell

W. Chan, Navdeep Jaitly, Quoc V. Le, Oriol Vinyals - 2015

4 papers in library cite

Speech recog + attention

[24]On Using monolingual corpora in Neural Machine Translation

C. G. Gulcehre, O. Firat, K. Xu, Kyunghyun Cho, L. Barrault, H. C. Lin, F. Bougares, Holger Schwenk, Yoshua Bengio - 2015

3 papers in library cite

Monolingual dataset for MT?

[25]Findings of the 2015 Workshop on Statistical Machine Translation

O. Bojar, R. Chatterjee, C. Federmann, B. Haddow, M. Huck, C. Hokamp, P. Koehn, V. Logacheva, C. Monz, M. Negri, M. Post, C. Scarton, L. Specia, M. Turchi - 2015

3 papers in library cite

[26]Montreal Neural Machine Translation Systems for WMT'15

S. Jean, O. Firat, Kyunghyun Cho, R. Memisevic, Yoshua Bengio - 2015

3 papers in library cite

[27]Annotated Gigaword

C. Napoles, M. Gormley, B. V. Durme - 2012

2 papers in library cite

[28]Semi-Supervised Learning for Neural Machine Translation

Y. Cheng, Weixin Xu, Z. He, Weiran He, H. Wu, Maosong Sun, Yibo Liu - 2016

2 papers in library cite

[29]Transfer Learning for Low-Resource Neural Machine Translation

Barret Zoph, Deniz Yuret, J. May, K. Knight - 2016

2 papers in library cite

[30]Zero-Resource Translation With Multi-Lingual Neural Machine Translation

O. Firat, B. Sankaran, Y. A. Onaizan, F. T. Y. Vural, Kyunghyun Cho - 2016

2 papers in library cite

[31]Achieving Human Parity in Conversational Speech Recognition

W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, Andreas Stolcke, D. Yu, Geoffrey Zweig - 2016

1 paper in library cites

[32]Asynchronous Translations With Recurrent Neural Nets

R. P. Neco, M. L. Forcada - 1997

1 paper in library cites

[33]Edinburgh's Statistical Machine Translation Systems for Wmt16

P. Williams, R. Sennrich, M. Nadejde, M. Huck, B. Haddow, O. Bojar - 2016

1 paper in library cites

[34]Exploiting Source-Side Monolingual Data in Neural Machine Translation

J. Zhang, C. Zong - 2016

1 paper in library cites

[35]Improving LSTM-based Video Description With Linguistic Knowledge Mined From Text

S. Venugopalan, L. A. Hendricks, R. Mooney, K. Saenko - 2016

1 paper in library cites

[36]Several Studies on Natural Language and Back-Propagation

R. B. Allen - 1987

1 paper in library cites

[37]The Edit Distance Transducer in Action: The university of cambridge English-German System at Wmt16

F. Stahlberg, E. Hasler, B. Byrne - 2016

1 paper in library cites

[38]Unsupervised Learning of Predictors From Unpaired Input-Output Samples

Jixuan Chen, P. S. Huang, X. He, Jianfeng Gao, L. Deng - 2016

1 paper in library cites

[39]Very Deep Convolutional Networks for End-to-End Speech Recognition

Y. Z. Zhang, W. Chan, Navdeep Jaitly - 2016

1 paper in library cites

Cited by

9

papers in your library

Cites

24

papers in your library

Read

on October 30, 2025

It's alright, but it's the same Seq2Seq thing with pretraining

Tags

Paper Aliases

No aliases