Papperoni

2013

On the Difficulty of Training Recurrent Neural Networks

Razvan Pascanu, Tomas Mikolov, Yoshua Bengio

Open PDF Google Scholar

citations

Cite Score

81

AI summary

This paper explores the exploding and vanishing gradient problems in RNNs, introduces a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. The paper validates the proposed solutions on pathological synthetic datasets and on polyphonic music prediction and language modeling tasks.

Main Contributions

The paper analyzes the exploding and vanishing gradient problems in RNNs from an analytical, a geometric and a dynamical systems perspective.
It proposes a gradient norm clipping strategy to deal with exploding gradients.
It proposes a soft constraint for the vanishing gradients problem.
It validates empirically the hypothesis and proposed solutions on the experimental section.
The paper shows that the proposed solutions improve performance on both the pathological synthetic datasets considered as well as on polyphonic music prediction and language modelling.

Abstract

There are two widely known issues with properly training Recurrent Neural Networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.

Citation Graph

Loading graph...

References [25]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Learning Representations by Back-Propagating Errors

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

34 papers in library cite

Introduced backprop. Short and simple.

[3]Finding Structure in Time

Jeffrey L. Elman - 1990

23 papers in library cite

Good paper overall that introduces the concept of an RNN. However, applications and results are still very primitive.

[4]Adaptive Subgradient Methods for Online Learning and Stochastic Optimization

John Duchi, Elad Hazan, Yoram Singer - 2011

19 papers in library cite

I actually skimmed through most of this. It's not a bad paper, but it's a math paper, not AI.

[5]Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi - 1994

31 papers in library cite

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

[6]Generating Text With Recurrent Neural Networks

Ilya Sutskever, James Martens, Geoffrey E. Hinton - 2011

13 papers in library cite

Pleasant paper but results are underwhelming. They use RNNs for character-level modeling, which is different. They also use the hessian-free method proposed by Martens, but don't go too deep into how it works, which is nice because otherwise it would be very mathy. Other papers cite this more as an example of usage rather than an actual milestone.

[7]Theano: New Features and Speed Improvements

F. Bastien, P. Lamblin, Razvan Pascanu, James Bergstra, I. Goodfellow, A. Bergeron, A. Bouchard, N. Nicolas, Yoshua Bengio - 2012

13 papers in library cite

The paper itself is ok, but it seems like they just wanted to make a paper to say "hey, ours is so much better now after these improvements, we crushed Torch7". A bit tryhard tbh.

[8]Generalization of Backpropagation With Application to a Recurrent Gas Market Model

Paul J. Werbos - 1988

11 papers in library cite

The first two sections are amazing. Very nice concepts, well explained. From section 3 on it gets more abstract and too formalized, and it's mostly for Hopfield recurrent nets (so not very relevant now). Section 4 is just the application to the gas market, which I don't care for at all.

[9]Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Pascal Vincent - 2012

8 papers in library cite

I think the main contribution is using HF optimization very early on, but very boring use case and they mix RNNs with RBMs which makes this soooo complicated.

[10]Learning Recurrent Neural Networks With Hessian-Free Optimization

James Martens, Ilya Sutskever - 2011

13 papers in library cite

Meh, very very mathy and seems like a minor improvement on LSTMs. They give very impressive results but I don't think it's too much of a fair comparison (vs. a 1999 architecture of LSTMs). I think they did well in showing that exploding/vanishing gradients can be overcome, but their method is not "it". As said by Bengio, they are partially responsible for the revival of RNNs (alongside Mikolov)

[11]Advances in Optimizing Recurrent Networks

Yoshua Bengio, N. B. Lewandowski, Razvan Pascanu - 2013

4 papers in library cite

Amazing paper that summarizes basically everything that was happening up to 2012 and explains why there was a boom in RNN in 2013 onwards.

[12]Empirical Evaluation and Combination of Advanced Language Modeling Techniques

Tomas Mikolov, A. Deoras, S. Kombrink, Lukas Burget, Jan Cernocky - 2011

13 papers in library cite

Early work proving that NNs can be good. But very uninteresting overall.

[13]Subword Language Modeling With Neural Networks

Tomas Mikolov, Ilya Sutskever, A. Deoras, H. S. Le, S. Kombrink, Jan Cernocky - 2012

7 papers in library cite

Doesn't add a ton of contribution other than saying that subword models can perform better than char-level models. Probably the most important thing here is early use of subwords.

[14]Theano: A CPU and GPU Math Expression Compiler

James Bergstra, O. Breuleux, F. Bastien, P. Lamblin, Razvan Pascanu, G. Desjardins, J. Turian, D. W. Farley, Yoshua Bengio - 2010

22 papers in library cite

Very nice framework. Symbolic programming is very nice. However, I think that this had very little impact and was mostly used by Bengio's lab.

[15]Statistical Language Models Based on Neural Networks

Tomas Mikolov - 2012

17 papers in library cite

Mikolov's Thesis

[16]A Novel Connectionist System for Unconstrained Handwriting Recognition

Alex Graves, M. Liwicki, Santiago Fernandez, R. Bertolami, H. Bunke, Jürgen Schmidhuber - 2009

5 papers in library cite

[17]Optimization and Applications of Echo State Networks With Leaky- Integrator Neurons

Herbert Jaeger, M. Lukosevicius, D. Popovici, U. Siewert - 2007

3 papers in library cite

[18]Reservoir Computing Approaches to Recurrent Neural Network Training

M. Lukosevicius, Herbert Jaeger - 2009

2 papers in library cite

[19]The Problem of Learning Long-Term Dependencies in Recurrent Networks

Yoshua Bengio, Paolo Frasconi, Patrice Simard - 1993

2 papers in library cite

[20]A Neurodynamical Model for Working Memory

Razvan Pascanu, Herbert Jaeger - 2011

1 paper in library cites

[21]Adaptive Synchronization of Neural and Physical Oscillators

K. Doya, S. Yoshizawa - 1991

1 paper in library cites

[22]Bifurcations of Recurrent Neural Networks in Gradient Descent Learning

K. Doya - 1993

1 paper in library cites

[23]Long Short-Term Memory in Echo state Networks: Details of a Simulation Study

Herbert Jaeger - 2012

1 paper in library cites

[24]Neural Networks With Adaptive Learning Rate and Momentum Terms

M. Moreira, E. Fiesler - 1995

1 paper in library cites

[25]Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering (Studies in Nonlinearity)

S. Strogatz - 1994

1 paper in library cites

Cited by

21

papers in your library

Cites

14

papers in your library

Read

on June 20, 2025

It starts very mathy but in the end there are some very nice contributions! You don't actually need to understand the math to know what's going on in the end.

Tags

Paper Aliases

No aliases