Papperoni

2007

Training Recurrent Networks by EVOLINO

Jürgen Schmidhuber, Daan Wierstra, Matteo Gagliolo, F. J. Gomez

Open PDF Google Scholar

citations

Cite Score

16

AI summary

This paper introduces Evolino, a novel method for training recurrent neural networks by evolving weights to hidden nodes and using linear methods for optimal linear mappings, demonstrating its ability to solve tasks unlearnable by Echo State nets and outperform gradient descent RNNs.

Main Contributions

Introduces Evolino, a novel method for training RNNs that combines neuroevolution for hidden unit weights and analytical linear methods for output mappings.
Presents PI-Evolino, which uses pseudoinverse-based linear regression to minimize summed squared error for output weights.
Presents Evoke, which uses quadratic programming to maximize the margin, creating the first evolutionary recurrent support vector machines.
Demonstrates that Evolino-based LSTM can solve context-sensitive language tasks and multiple superimposed sine wave generation tasks that Echo State Networks (ESNs) cannot.
Achieves higher accuracy in continuous function generation tasks compared to conventional gradient descent RNNs, including gradient-based LSTM, and generalizes well from minimal training data.

Abstract

In recent years, gradient-based LSTM recurrent neural networks (RNNs) solved many previously RNN-unlearnable tasks. Sometimes, however, gradient information is of little use for training RNNs, due to numerous local minima. For such cases, we present a novel method: EVOlution of systems with LINear Outputs (Evolino). Evolino evolves weights to the nonlinear, hidden nodes of RNNs while computing optimal linear mappings from hidden state to output, using methods such as pseudo-inverse-based linear regression. If we instead use quadratic programming to maximize the margin, we obtain the first evolutionary recurrent support vector machines. We show that Evolino-based LSTM can solve tasks that Echo State nets (Jaeger, 2004a) cannot and achieves higher accuracy in certain continuous function generation tasks than conventional gradient descent RNNs, including gradient-based LSTM.

Citation Graph

Loading graph...

References [54]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Learning to Forget: Continual Prediction With LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins - 2000

13 papers in library cite

This defined what an LSTM is. Before that the LSTM didn't know how to forget. Very nice.

[3]Harnessing Nonlinearity: Predicting Chaotic Systems and Saving Energy in Wireless Communication

Herbert Jaeger, Harald Haas - 2004

4 papers in library cite

Bad read. Doesn't explain anything of how things work. It's boring and I didn't like the figures. Maybe the main paper about ESNs is better, but frankly I don't want to read it.

[4]Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber - 2001

16 papers in library cite

Wow, this is so much better than the other paper - I should have read it sooner. It's concise and not too abstract, and also gives very good context on RNN problems and how to solve them.

[5]Generalization of Backpropagation With Application to a Recurrent Gas Market Model

Paul J. Werbos - 1988

11 papers in library cite

The first two sections are amazing. Very nice concepts, well explained. From section 3 on it gets more abstract and too formalized, and it's mostly for Hopfield recurrent nets (so not very relevant now). Section 4 is just the application to the gas market, which I don't care for at all.

[6]LSTM Can Solve Hard Long Time Lag Problems

Sepp Hochreiter, Jürgen Schmidhuber - 1997

5 papers in library cite

Doesn't add much on top of what we already know. Good paper nonetheless.

[7]Evaluating Benchmark Problems by Random Guessing

Jürgen Schmidhuber, Sepp Hochreiter, Yoshua Bengio - 2001

1 paper in library cites

This could be 3 pages long. They just randomly select the weights of the network. The rest is just description of the experiments. Should be a 2 but I am giving them credit for showing that RG can work.

[8]Framewise Phoneme Classification With Bidirectional LSTM and Other Neural Network Architectures

Alex Graves, Jürgen Schmidhuber - 2005

14 papers in library cite

Very nice paper! Simple, no bullshit. Just "hey, we have LSTM and we have BRNN, let's try to join it"

[9]Parallel Distributed Processing

D. E. Rumelhart, J. L. Mcclelland, P. R. Group - 1986

15 papers in library cite

[10]Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences

P. Werbos - 1974

14 papers in library cite

Werbos PhD Thesis, introduced backprop

[11]Untersuchungen zu dynamischen neuronalen netzen

Sepp Hochreiter - 1991

18 papers in library cite

[12]The Utility Driven Dynamic Error Propagation Network

A. J. Robinson, F. Fallside - 1987

10 papers in library cite

[13]Learning Precise Timing With LSTM Recurrent Networks

F. Gers, N. Schraudolph, Jürgen Schmidhuber - 2002

9 papers in library cite

[14]The Nature of Statistical Learning Theory

V. Vapnik - 1995

9 papers in library cite

Book, 300+ pages

[15]Complexity of Exact Gradient Computation Algorithms for Recurrent Neural Networks

Ronald J. Williams - 1989

6 papers in library cite

[16]Gradient Calculations for Dynamic Recurrent Neural Networks: A Survey

B. A. Pearlmutter - 1995

5 papers in library cite

[17]A Fixed Size Storage O(n³) Time Complexity Learning algorithm for Fully Recurrent Continually Running Networks

Jürgen Schmidhuber - 1992

4 papers in library cite

[18]Adaptation in Natural and Artificial Systems

J. H. Holland - 1975

4 papers in library cite

[19]LSTM recurrent Networks Learn Simple Context-Free and Context-Sensitive Languages

Felix A. Gers, Jürgen Schmidhuber - 2001

3 papers in library cite

[20]Evolution and Optimum Seeking

H. P. Schwefel - 1995

2 papers in library cite

[21]Evolutionsstrategie—Optimierung technischer Systeme nach Prinzipien der biologischen Evolution

I. Rechenberg - 1973

2 papers in library cite

[22]Kalman Filters Improve LSTM Network Performance in Problems Un- Solvable by Traditional Recurrent Nets

J. A. P. Ortiz, F. Gers, D. Eck, Jürgen Schmidhuber - 2002

2 papers in library cite

[23]Predicting Time Series With Support Vector Machines

K. Muller, A. Smola, G. Ratsch, B. Scholkopf, J. Kohlmorgen, V. Vapnik - 1997

2 papers in library cite

[24]The echo State Approach to Analysing and Training Recurrent Neural Networks

Herbert Jaeger - 2001

2 papers in library cite

Echo state networks

[25]A Fresh Look at Real-Time Computation in Generic Recurrent Neural Circuits

W. Maass, T. Natschlager, H. Markram - 2002

1 paper in library cites

[26]A Generalized Inverse for Matrices

R. Penrose - 1955

1 paper in library cites

[27]A Review of Evolutionary Artificial Neural Networks

Xingcheng Yao - 1993

1 paper in library cites

[28]Co-Evolving Recurrent Neurons Learn Deep Memory POMDPS

F. J. Gomez, Jürgen Schmidhuber - 2005

1 paper in library cites

[29]Designing Neural Networks Using Genetic Algorithms

G. Miller, P. Todd, S. Hedge - 1989

1 paper in library cites

[30]Dynamic Time-Alignment Kernel in Support Vector Machine

H. Shimodaira, K. I. Noma, M. Nakai, S. Sagayama - 2002

1 paper in library cites

[31]Dynamische Neuronale Netze und das Fundamentale raumzeitliche Lernproblem

Jürgen Schmidhuber - 1990

1 paper in library cites

[32]Efficient Reinforcement Learning Through Symbiotic Evolution

D. E. Moriarty, R. Miikkulainen - 1996

1 paper in library cites

[33]Evolino for Recurrent Support Vector Machines

Jürgen Schmidhuber, Matteo Gagliolo, Daan Wierstra, Faustino Gomez - 2006

1 paper in library cites

[34]Evolino: Hybrid Neuroevolution/Optimal Linear Search for Sequence Prediction

Jürgen Schmidhuber, Daan Wierstra, F. J. Gomez - 2005

1 paper in library cites

Pretty much the same as the other one

[35]Evolving Mobile Robots in Simulated and Real Environments

O. Miglino, H. Lund, S. Nolfi - 1995

1 paper in library cites

[36]Evolving Modular Fast-Weight Networks for Control

Faustino Gomez, Jürgen Schmidhuber - 2005

1 paper in library cites

[37]Evolving Neural Networks With Collaborative Species

M. A. Potter, K. A. D. Jong - 1995

1 paper in library cites

[38]Evolving Virtual Creatures

K. Sims - 1994

1 paper in library cites

[39]Finding Good Echo State Networks to Control an Underwater Robot Using Evolutionary Computations

T. V. D. Zant, V. Becanovic, K. Ishii, H. U. Kobialka, P. G. Ploger - 2004

1 paper in library cites

[40]Framewise Phone Classification Using Support Vector Machines

J. Salomon, S. King, M. Osborne - 2002

1 paper in library cites

[41]How to Evolve Autonomous Robots: Different Approaches in Evolutionary Robotics

S. Nolfi, D. Floreano, O. Miglino, F. Mondada - 1994

1 paper in library cites

[42]Identification of Motion With echo State Network

K. Ishii, T. V. D. Zant, V. Becanovic, P. G. Ploger - 2004

1 paper in library cites

[43]Learning Nonregular Languages: A Comparison of Simple Recurrent Networks and LSTM

Jürgen Schmidhuber, F. Gers, D. Eck - 2002

1 paper in library cites

[44]Modeling Non-Linear Dynamical Systems With Evolino

Daan Wierstra, F. J. Gomez, Jürgen Schmidhuber - 2005

1 paper in library cites

[45]Nonlinear Prediction of Chaotic Time Series Using Support Vector Machines

S. Mukherjee, E. Osuna, F. Girosi - 1997

1 paper in library cites

[46]Numerische Optimierung von Computer-Modellen

H. P. Schwefel - 1977

1 paper in library cites

[47]Oscillation and Chaos in Physiological Control Systems

M. C. Mackey, L. Glass - 1977

1 paper in library cites

[48]RBF neural Network, Basis Functions and Genetic Algorithms

E. P. Maillard, D. Gueriot - 1997

1 paper in library cites

[49]Recurrent Least Squares Support Vector Machines

J. Suykens, J. Vandewalle - 2000

1 paper in library cites

[50]Robust Nonlinear Control Through Neuroevolution

F. J. Gomez - 2003

1 paper in library cites

[51]Sequential Behavior and Learning in Evolved Dynamical Neural Networks

B. M. Yamauchi, R. D. Beer - 1994

1 paper in library cites

[52]Solving Non-Markovian Control Tasks With Neuroevolution

Faustino Gomez, R. Miikkulainen - 1999

1 paper in library cites

[53]Symbiotic Evolution of Neural Networks in Sequential Decision Tasks

D. E. Moriarty - 1997

1 paper in library cites

[54]Turing Computability With Neural Nets

H. T. Siegelmann, E. D. Sontag - 1991

1 paper in library cites

Cited by

1

papers in your library

Cites

10

papers in your library

Read

on January 29, 2026

Very mid. They use toy tasks and it's really unclear if Evolino is better than just SGD. Nice variation on learning networks though

Tags

Paper Aliases

No aliases