Papperoni

2015

A Theoretically Grounded Application of Dropout in Recurrent Neural Networks

Yarin Gal

Open PDF Google Scholar

citations

Cite Score

57

AI summary

This paper introduces a new variational inference based dropout technique applied to LSTM and GRU models, achieving state-of-the-art results on language modeling with the Penn Treebank dataset and improving performance on sentiment analysis tasks. The technique outperforms existing methods and offers insights into using dropout with RNN models.

Main Contributions

Proposes a new variational inference based dropout technique for RNNs.
Applies the technique to LSTM and GRU models.
Achieves state-of-the-art results on language modeling with the Penn Treebank dataset (73.4 test perplexity).
Improves performance on sentiment analysis tasks.
Provides a theoretical grounding for dropout in approximate Bayesian inference for RNNs.

Abstract

Recurrent neural networks (RNNs) stand at the forefront of many recent developments in deep learning. Yet a major difficulty with these models is their tendency to overfit, with dropout shown to fail when applied to recurrent layers. Recent results at the intersection of Bayesian modelling and deep learning offer a Bayesian interpretation of common deep learning techniques such as dropout. This grounding of dropout in approximate Bayesian inference suggests an extension of the theoretical results, offering insights into the use of dropout with RNN models. We apply this new variational inference based dropout technique in LSTM and GRU models, assessing it on language modelling and sentiment analysis tasks. The new approach outperforms existing techniques, and to the best of our knowledge improves on the single model state-of-the-art in language modelling with the Penn Treebank (73.4 test perplexity). This extends our arsenal of variational tools in deep learning.

Citation Graph

Loading graph...

References [30]

Sort:

Filter:

[1]Adam: A Method for Stochastic Optimization

D. P. Kingma, Jimmy Lei Ba - 2014

49 papers in library cite

Amazing paper! Very well explained and huge impact. I am amazed that they made something so simple even when it requires a lot of background mathematical knowledge

[2]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[3]Dropout: A Simple Way to Prevent Neural Networks From Overfitting

N. Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2014

20 papers in library cite

Good paper, but it's mostly a review of the method described in the other paper with more results. It's longer as well, so I would suggest just reading the other one.

[4]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[5]Sequence to Sequence Learning With Neural Networks

Ilya Sutskever, Oriol Vinyals, Quoc V. Le - 2014

58 papers in library cite

Good paper, but I think it only got famous because they set a new good baseline for NNs in MT. Their main contribution was reversing the source sentence TBH.

[6]Speech Recognition With Deep Recurrent Neural Networks

Geoffrey Hinton - 2013

13 papers in library cite

Small, simple, nice results. First usage of deep BiLSTM?

[7]Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors

Geoffrey E. Hinton, N. Srivastava, Alex Krizhevsky, Ilya Sutskever, Ruslan R. Salakhutdinov - 2012

25 papers in library cite

Dropout, super impactful. The idea that you are training many estimators at once is also very nice.

[8]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[9]LSTM neural Networks for Language Modeling

M. Sundermeyer, R. Schluter, Hermann Ney - 2010

7 papers in library cite

It reads like an undergrad project - doesn't add anything new.

[10]Recurrent Continuous Translation Models

N. Kalchbrenner, Phil Blunsom - 2013

27 papers in library cite

Good paper, probably the first that used an encoder-decoder. But they used a conv. NN instead of a tradicional decoder, which I don't really like.

[11]Theano: A CPU and GPU Math Expression Compiler

James Bergstra, O. Breuleux, F. Bastien, P. Lamblin, Razvan Pascanu, G. Desjardins, J. Turian, D. W. Farley, Yoshua Bengio - 2010

22 papers in library cite

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[12]Weight Uncertainty in Neural Network

C. Blundell, J. Cornebise, Koray Kavukcuoglu, Daan Wierstra - 2015

3 papers in library cite

Introduced Bayesian NNs. I am not tooo excited about it but sounds relevant

[13]Dropout Improves Recurrent Neural Networks for Handwriting Recognition

V. Pham, T. Bluche, C. Kermorvant, J. Louradour - 2014

5 papers in library cite

Dropout for RNNs

[14]Seeing Stars: Exploiting Class Relationships for Sentiment Categorization With Respect to Rating Scales

Bo Pang, L. Lee - 2005

13 papers in library cite

[15]Practical Variational Inference for Neural Networks

Alex Graves - 2011

8 papers in library cite

[16]Bayesian Learning for Neural Networks

R. Neal - 1995

3 papers in library cite

[17]Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Yarin Gal, Zoubin Ghahramani - 2015

3 papers in library cite

[18]Keeping the Neural Networks Simple by Minimizing the Description Length of the Weights

Geoffrey E. Hinton, D. V. Camp - 1993

3 papers in library cite

[19]On Fast Dropout and Its Applicability to Recurrent Networks

J. Bayer - 2013

3 papers in library cite

[20]Stochastic Backpropagation and Approximate Inference in Deep Generative Models

D. J. Rezende, S. Mohamed, Daan Wierstra - 2014

3 papers in library cite

[21]A Practical Bayesian Framework for Backpropagation Networks

D. J. Mackay - 1992

2 papers in library cite

[22]Probabilistic Backpropagation for Scalable Learning of Bayesian neural Networks

J. M. H. Lobato, R. A. Adams - 2015

2 papers in library cite

[23]Regularization and Nonlinearities for Neural Language Models: When Are They Needed?

M. Pachitariu, M. Sahani - 2013

2 papers in library cite

[24]Variational dropout and the Local Reparameterization Trick

D. Kingma, T. Salimans, M. Welling - 2015

2 papers in library cite

[25]Bayesian convolutional neural Networks With Bernoulli approximate Variational Inference

Yarin Gal, Zoubin Ghahramani - 2015

1 paper in library cites

[26]Bayesian dark Knowledge

A. K. Balan, V. Rathod, Kevin P. Murphy, M. Welling - 2015

1 paper in library cites

[27]Ensemble Learning in Bayesian neural Networks

D. Barber, C. M. Bishop - 1998

1 paper in library cites

Fchollet - 2015

1 paper in library cites

[29]RnnDrop: A Novel Dropout for RNNs in ASR

T. Moon, H. Choi, Honglak Lee, I. Song - 2015

1 paper in library cites

[30]Where to Apply Dropout in Recurrent Neural Networks for Handwriting Recognition?

T. Bluche, C. Kermorvant, J. Louradour - 2015

1 paper in library cites

Cited by

9

papers in your library

Cites

13

papers in your library

Read

on December 25, 2025

The methodology is very simple and effective, but maybe it was too simple for a paper, so they put A TON of math on top of it... Unnecessary really.

Tags

Paper Aliases

No aliases