Papperoni

1994

Learning Long-Term Dependencies With Gradient Descent Is Difficult

Yoshua Bengio, Patrice Simard, Paolo Frasconi

Open PDF Google Scholar

citations

Cite Score

88

AI summary

This paper investigates the difficulties of training recurrent neural networks with gradient descent for tasks with long-term dependencies; it introduces a minimal task with a recurrent neuron and demonstrates that gradient descent fails for long-term dependencies, highlighting a trade-off between learning and information latching.

Main Contributions

Demonstrates the difficulty of training recurrent neural networks to learn long-term dependencies using gradient descent.
Introduces the concept of information latching as a requirement for storing state information over arbitrary durations.
Presents experimental results on a minimal task, showing that gradient descent fails even in simple cases.
Provides theoretical analysis linking the problem to vanishing gradients when using hyperbolic attractors for robust information storage.
Suggests alternative approaches to address the vanishing gradient problem, including simulated annealing, multi-grid random search, and time-weighted pseudo-Newton optimization.

Abstract

Abstract- Recurrent neural networks can be used to map input sequences to output sequences, such as for recognition, production or prediction problems. However, practical difficulties have been reported in training recurrent neural networks to perform tasks in which the temporal contingencies present in the input/output sequences span long intervals. We show why gradient based learning algorithms face an increasingly difficult problem as the duration of the dependencies to be captured increases. These results expose a trade-off between efficient learning by gradient descent and latching on information for long periods. Based on an understanding of this problem, alternatives to standard gradient descent are considered.

Citation Graph

Loading graph...

References [23]

Sort:

Filter:

[1]Learning Internal Representations by Error Propagation

D. E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams - 1986

46 papers in library cite

I expected very little of this, but was so good in explaining concepts! Very good read. It gets a bit boring when it starts explaining things by the end of the chapter, but good nonetheless.

[2]Improving the Convergence of Back-Propagation Learning With Second-Order Methods

S. Becker, Yann Lecun - 1988

9 papers in library cite

Surprisingly good. I thought this would be very math-heavy but in the end it's very easy to understand. Sadly results are underwhelming

[3]A Learning algorithm for Continually Running Fully Recurrent Neural Networks

R. Williams, David Zipser - 1989

8 papers in library cite

Quick read and very simple concept. I wish they explained things using a more visual approach instead of maths, but it's ok to follow nonetheless.

[4]Learning Processes in an Asymmetric Threshold Network

Yann Lecun - 1986

3 papers in library cite

Meh. Really did not bring much to the table. Describes back-prop but mentions that it comes from Hinton.

[5]Optimization by Simulated Annealing

S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi - 1983

6 papers in library cite

[6]Induction of Multiscale Temporal Structure

M. C. Mozer - 1992

5 papers in library cite

[7]A First Look at Phonetic Discrimination Using a Connectionist Architecture With Recurrent Links

G. Kuhn - 1987

3 papers in library cite

[8]A Focused Back-Propagation Algorithm for Temporal Pattern Recognition

M. C. Mozer - 1989

3 papers in library cite

[9]Global Optimization of a Neural Network - Hidden Markov Model Hybrid

Yoshua Bengio, R. D. Mori, G. Flammia, F. Kompe - 1991

2 papers in library cite

[10]Iterative Solution of Non-Linear Equations in Several Variables and Systems

J. Ortega, W. Rheinboldt - 1970

2 papers in library cite

[11]Learning by Choice of Internal Representations

T. Grossman, R. Meir, E. Domany - 1989

2 papers in library cite

[12]The 'Moving Targets' Training algorithm

R. Rohwer - 1990

2 papers in library cite

[13]The Development of the Time-Delay Neural Network Architecture for Speech Recognition

Kevin J. Lang, Geoffrey E. Hinton - 1988

2 papers in library cite

[14]The Problem of Learning Long-Term Dependencies in Recurrent Networks

Yoshua Bengio, Paolo Frasconi, Patrice Simard - 1993

2 papers in library cite

[15]A Method of Training Multi-Layer Networks With Heaviside Characteristics Using Internal Representations

R. J. Gaynier, T. Downs - 1993

1 paper in library cites

[16]Artificial Neural Networks and Their Application to Sequence Recognition

Yoshua Bengio - 1991

1 paper in library cites

[17]BPS: A Learning algorithm for Capturing the Dynamic Nature of Speech

M. Gori, Yoshua Bengio, R. D. Mori - 1989

1 paper in library cites

Missing year

[18]Inserting Rules Into Recurrent Neural Networks

C. L. Giles, C. W. Omlin

1 paper in library cites

[19]Local Feedback Multilayered Networks

Paolo Frasconi, M. Gori, G. Soda - 1992

1 paper in library cites

[20]Minimizing Multimodal Functions of Continuous Variables With the Simulated Annealing algorithm

A. Corana, M. Marchesi, C. Martini, S. Ridella - 1987

1 paper in library cites

[21]Nonlinear Dynamics and Stability of Analog Neural Networks

C. M. Marcus, F. R. Waugh, R. M. Westervelt - 1991

1 paper in library cites

Missing year

[22]Unified Integration of Explicit Rules and Learning by Example in Recurrent Networks

Paolo Frasconi, M. Gori, M. Maggini, G. Soda

1 paper in library cites

[23]Using Random Weights to Train Multi-Layer Networks of Hard-Limiting Units

P. L. Bartlett, T. Downs - 1992

1 paper in library cites

Cited by

31

papers in your library

Cites

4

papers in your library

Read

on April 25, 2025

The first ones to notice that there is a problem with gradient descent, but way too mathy for me.

Tags

Paper Aliases

No aliases