Papperoni

2014

Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals

Open PDF Google Scholar

citations

Cite Score

70

AI summary

This paper introduces a method to correctly apply dropout to LSTMs, reducing overfitting in language modeling, speech recognition, image caption generation, and machine translation, achieving improved results. It shows that standard dropout perturbs recurrent connections, which makes it difficult for the LSTM to learn.

Main Contributions

Introduces a method to correctly apply dropout to LSTMs
Dropout is only applied to the non-recurrent connections
Demonstrates that the method reduces overfitting on a variety of tasks
Achieves improved results on language modeling, speech recognition, image caption generation, and machine translation

Abstract

We present a simple regularization technique for Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units. Dropout, the most successful technique for regularizing neural networks, does not work well with RNNS and LSTMs. In this paper, we show how to correctly apply dropout to LSTMs, and show that it substantially reduces overfitting on a variety of tasks. These tasks include language modeling, speech recognition, image caption generation, and machine translation.

Citation Graph

Loading graph...

References [34]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Going Deeper With Convolutions

Christian Szegedy, Weizhou Liu, Y. Jia, P. Sermanet, S. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich - 2015

20 papers in library cite

Introduced the inception algorithm, which is nice. The paper is quite good, but I had to google some stuff to understand it fully. Nice contribution and SotA, but TBH I felt that it wasn't toooo good of a read.

[3]Microsoft COCO: Common Objects in Context

T. Y. Lin, M. Maire, S. Belongie, James Hays, Pietro Perona, D. Ramanan, Piotr Dollar, C. L. Zitnick - 2014

14 papers in library cite

I liked this paper a lot. It's a bit long and I was already a bit tired, but it was nice overall.

[4]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[5]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[6]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

Small, simple, nice results. First usage of deep BiLSTM?

[7]Building a Large Annotated Corpus of English: The Penn Treebank

M. P. Marcus, B. Santorini, Mary Ann Marcinkiewicz - 1993

22 papers in library cite

Well, not really interesting but very cool to see how the peen tree bank was made.

[8]Show and Tell: A Neural Image Caption Generator

Dumitru Erhan - 2015

11 papers in library cite

It's nice and they beat a ton of SotA. However, I read the one that uses attention first so this is a bit less surprising.

[9]Recurrent Neural Network Based Language Model

Tomas Mikolov, M. Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur - 2010

36 papers in library cite

The comeback of RNNs for language modeling. Not too exciting but impactful and a short read.

[10]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[11]Regularization of Neural Networks Using Dropconnect

L. Wan, M. Zeiler, S. Zhang, Rob Fergus - 2013

8 papers in library cite

I feel that the method is very complex and does not improve much on top of regular dropout.

[12]LSTM neural Networks for Language Modeling

M. Sundermeyer, R. Schluter, Hermann Ney - 2010

7 papers in library cite

It reads like an undergrad project - doesn't add anything new.

[13]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[14]Exploiting Similarities Among Languages for Machine Translation

Tomas Mikolov, Quoc V. Le, Ilya Sutskever - 2013

6 papers in library cite

It's just word2vec + a projection, but results are very nice and it's surprisingly simple!

[15]How to Construct Deep Recurrent Neural Networks

Razvan Pascanu, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2013

7 papers in library cite

Very interesting despite not being too relevant. Good read and a new way of thinking about RNNs.

[16]Context Dependent Recurrent Neural Network Language Model

Tomas Mikolov, Geoffrey Zweig - 2012

12 papers in library cite

Nothing too interesting, just using the context of the RNN.

[17]Strategies for Training Large Scale Neural Network Language Models

Tomas Mikolov, A. Deoras, D. Povey, Lukas Burget, Jan Cernocky - 2011

9 papers in library cite

Just builds on other things. Very minor suff in my opinion.

[18]Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, John Makhoul - 2014

9 papers in library cite

Simple and very nice that it showed that NNs can stand on their own for NMT.

[19]Dropout Improves Recurrent Neural Networks for Handwriting Recognition

V. Pham, T. Bluche, C. Kermorvant, J. Louradour - 2014

5 papers in library cite

Dropout for RNNs

[20]A Clockwork RNN

J. Koutnik, K. Greff, Faustino Gomez, Jürgen Schmidhuber - 2014

4 papers in library cite

Interesting title, not well cited... I added because it's schmidhuber

[21]Statistical Language Models Based on Neural Networks

Tomas Mikolov - 2012

17 papers in library cite

Mikolov's Thesis

[22]Connectionist Speech Recognition: A Hybrid Approach

H. Bourlard, N. Morgan - 1993

8 papers in library cite

[23]Improving Neural Networks With Dropout

N. Srivastava - 2013

6 papers in library cite

[24]A Novel Connectionist System for Unconstrained Handwriting Recognition

Alex Graves, M. Liwicki, Santiago Fernandez, R. Bertolami, H. Bunke, Jürgen Schmidhuber - 2009

5 papers in library cite

[25]Fast Dropout Training

Shijie Wang, C. Manning - 2013

4 papers in library cite

[26]On Fast Dropout and Its Applicability to Recurrent Networks

J. Bayer - 2013

3 papers in library cite

[27]Optimization and Applications of Echo State Networks With Leaky- Integrator Neurons

Herbert Jaeger, M. Lukosevicius, D. Popovici, U. Siewert - 2007

3 papers in library cite

[28]Byblos: The BBN Continuous Speech Recognition System

Y. Chow, M. Dunham, O. Kimball, M. Krasner, G. Kubala, John Makhoul, P. Price, S. Roucos, Richard Schwartz - 1987

2 papers in library cite

[29]Language Modeling With Sum-Product Networks

W. C. Cheng, S. Kok, H. V. Pham, H. L. Chieu, K. M. A. Chai - 2014

2 papers in library cite

[30]Regularization and Nonlinearities for Neural Language Models: When Are They Needed?

M. Pachitariu, M. Sahani - 2013

2 papers in library cite

[31]The Use of Recurrent Neural Networks in Continuous Speech Recognition

Tony Robinson, M. Hochberg, S. Renals - 1996

2 papers in library cite

[32]University le mans

Holger Schwenk - 2014

2 papers in library cite

[33]Lium's SMT Machine Translation Systems for WMT 2011

Holger Schwenk, P. Lambert, L. Barrault, C. Servan, H. Afli, S. A. Rauf, K. Shah - 2011

1 paper in library cites

[34]Sequence Discriminative Distributed Training of Long Short-Term Memory Recurrent Neural Networks

H. Sak, Oriol Vinyals, Georg Heigold, A. Senior, E. Mcdermott, R. Monga, M. Mao - 2014

1 paper in library cites

Cited by

22

papers in your library

Cites

20

papers in your library

Read

on October 13, 2025

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

Tags

Paper Aliases

No aliases