Papperoni

2015

LSTM: A Search Space Odyssey

K. Greff, R. K. Srivastava, J. Koutn'ik, B. R. Steunebrink, Jürgen Schmidhuber

Open PDF Google Scholar

citations

Cite Score

82

AI summary

This paper presents a large-scale analysis of eight LSTM variants on speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the fANOVA framework.

Main Contributions

The paper presents the first large-scale analysis of eight LSTM variants on three representative tasks.
The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful fANOVA framework.
The results show that none of the variants can improve upon the standard LSTM architecture significantly.
The results demonstrate the forget gate and the output activation function to be the most critical components of the LSTM architecture.
The paper observes that the studied hyperparameters are virtually independent and derives guidelines for their efficient adjustment.

Abstract

Abstract—Several variants of the Long Short-Term Memory (LSTM) architecture for recurrent neural networks have been proposed since its inception in 1995. In recent years, these networks have become the state-of-the-art models for a variety of machine learning problems. This has led to a renewed interest in understanding the role and utility of various computational components of typical LSTM variants. In this paper, we present the first large-scale analysis of eight LSTM variants on three representative tasks: speech recognition, handwriting recognition, and polyphonic music modeling. The hyperparameters of all LSTM variants for each task were optimized separately using random search, and their importance was assessed using the powerful fANOVA framework. In total, we summarize the results of 5400 experimental runs (≈ 15 years of CPU time), which makes our study the largest of its kind on LSTM networks. Our results show that none of the variants can improve upon the standard LSTM architecture significantly, and demonstrate the forget gate and the output activation function to be its most critical components. We further observe that the studied hyperparameters are virtually independent and derive guidelines for their efficient adjustment.

Citation Graph

Loading graph...

References [45]

Sort:

Filter:

[1]Long Short-Term Memory

Sepp Hochreiter, Jürgen Schmidhuber - 1997

94 papers in library cite

LSTMs FTW!

[2]Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation

Kyunghyun Cho, B. V. Merrienboer, C. G. Gulcehre, D. Bahdanau, F. Bougares, Holger Schwenk, Yoshua Bengio - 2014

38 papers in library cite

Introduces RNN encoder-decoder. I love it :)

[3]Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

J. Chung, C. G. Gulcehre, Kyunghyun Cho, Yoshua Bengio - 2014

11 papers in library cite

It's a good paper but results are veeeery underwhelming.

[4]Random Search for Hyper-Parameter Optimization

James Bergstra, Yoshua Bengio - 2012

7 papers in library cite

It seems crazy that it was only in 2012 that they found out that random search was good! Still, kudos on them for noticing, and the paper is just so easy to follow and enjoyable!

[5]Learning to Forget: Continual Prediction With LSTM

Felix A. Gers, Jürgen Schmidhuber, Fred Cummins - 2000

13 papers in library cite

This defined what an LSTM is. Before that the LSTM didn't know how to forget. Very nice.

[6]Connectionist Temporal Classification: Labelling Unsegmented Sequence Data With Recurrent Neural Networks

Alex Graves, Santiago Fernandez, Faustino Gomez, Jürgen Schmidhuber - 2006

7 papers in library cite

It's a bit lukewarm. Nice idea but execution was a bit meh. I also think the prefix search was unnecessarily complex and loses to beam search (as they admit later on)

[7]On the Importance of Initialization and Momentum in Deep Learning

Ilya Sutskever, James Martens, G. Dahl, Geoffrey Hinton - 2013

13 papers in library cite

They give very good context and it's easy to understand that they are doing this as a counterpoint to HF. Surprising results as well. I just think it was made obsolete by relu

[8]Generating Sequences With Recurrent Neural Networks

Alex Graves - 2013

27 papers in library cite

Very cool and is the first to actually proposed the Attention mechanism! It gets a bit mathy but nothing too crazy. Also has the first examples of good machine generated writing I've seen in these papers, so very nice results.

[9]Recurrent Neural Network Regularization

Wojciech Zaremba, Ilya Sutskever, Oriol Vinyals - 2014

22 papers in library cite

It's a very simple idea and TBH it's nothing different from dropout. It's good that it's a very short paper and very straightforward, but could be a paragraph long.

[10]Gradient Flow in Recurrent Nets: The Difficulty of Learning Long-Term Dependencies

Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber - 2001

16 papers in library cite

Wow, this is so much better than the other paper - I should have read it sooner. It's concise and not too abstract, and also gives very good context on RNN problems and how to solve them.

[11]Generalization of Backpropagation With Application to a Recurrent Gas Market Model

Paul J. Werbos - 1988

11 papers in library cite

The first two sections are amazing. Very nice concepts, well explained. From section 3 on it gets more abstract and too formalized, and it's mostly for Hopfield recurrent nets (so not very relevant now). Section 4 is just the application to the gas market, which I don't care for at all.

[12]Recurrent Nets That Time and Count

Felix A. Gers, Jürgen Schmidhuber - 2000

1 paper in library cites

Not a whole lot different from regular LSTMs. A bit hard to read, and only had toy tasks. Low 3.

[13]Addressing the Rare Word Problem in Neural Machine Translation

T. Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, Wojciech Zaremba - 2014

14 papers in library cite

The method was very poorly explained. It was also worse than a paper released sooner, and more complicated. Overall not that good.

[14]Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription

Pascal Vincent - 2012

8 papers in library cite

I think the main contribution is using HF optimization very early on, but very boring use case and they mix RNNs with RBMs which makes this soooo complicated.

[15]Training Recurrent Networks by EVOLINO

Jürgen Schmidhuber, Daan Wierstra, Matteo Gagliolo, F. J. Gomez - 2007

1 paper in library cites

Very mid. They use toy tasks and it's really unclear if Evolino is better than just SGD. Nice variation on learning networks though

[16]Recent Advances in Finding Best Operating Conditions

R. L. Anderson - 1953

1 paper in library cites

The paper is not bad, it's just very abstract and no experiments. It's more documenting existing methods without giving much context, explanation, or proof. I also hated that it has nothing to do with Random Search (the reason I read it) - there's only a small mention about it in the end.

[17]Framewise Phoneme Classification With Bidirectional LSTM and Other Neural Network Architectures

Alex Graves, Jürgen Schmidhuber - 2005

14 papers in library cite

Very nice paper! Simple, no bullshit. Just "hey, we have LSTM and we have BRNN, let's try to join it"

[18]Long-Term Recurrent Convolutional Networks for Visual Recognition and Description

J. Donahue, L. Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, Trevor Darrell - 2014

4 papers in library cite

Early multimodality

[19]Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling

H. Sak, A. W. Senior, F. Beaufays - 2014

5 papers in library cite

Only 5 pages

[20]Dropout Improves Recurrent Neural Networks for Handwriting Recognition

V. Pham, T. Bluche, C. Kermorvant, J. Louradour - 2014

5 papers in library cite

Dropout for RNNs

[21]Minimization by Random Search Techniques

F. J. Solis, R. J. B. Wets - 1981

1 paper in library cites

Other random search OG

[22]Untersuchungen zu dynamischen neuronalen netzen

Sepp Hochreiter - 1991

18 papers in library cite

[23]The Utility Driven Dynamic Error Propagation Network

A. J. Robinson, F. Fallside - 1987

10 papers in library cite

[24]Practical Bayesian Optimization of Machine Learning Algorithms

J. Snoek, Hugo Larochelle, R. P. Adams - 2012

9 papers in library cite

[25]Complexity of Exact Gradient Computation Algorithms for Recurrent Neural Networks

Ronald J. Williams - 1989

6 papers in library cite

[26]Supervised Sequence Labelling With Recurrent Neural Networks

Alex Graves - 2012

6 papers in library cite

[27]A Novel Connectionist System for Unconstrained Handwriting Recognition

Alex Graves, M. Liwicki, Santiago Fernandez, R. Bertolami, H. Bunke, Jürgen Schmidhuber - 2009

5 papers in library cite

[28]The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus (TIMIT)

D. Isto - 1990

5 papers in library cite

[29]Unconstrained Online Handwriting Recognition With Recurrent Neural Networks

Alex Graves, Santiago Fernandez, M. Liwicki, H. Bunke, Jürgen Schmidhuber - 2008

5 papers in library cite

[30]An Empirical Exploration of Recurrent Network Architectures

R. Jozefowicz, Wojciech Zaremba, Ilya Sutskever - 2015

4 papers in library cite

[31]IAM-OnDB an on-Line English Sentence Database Acquired From Handwritten Text on a Whiteboard

M. Liwicki, H. Bunke - 2005

3 papers in library cite

[32]Sequential Model-Based Optimization for General Algorithm configuration

Frank Hutter, H. H. Hoos, K. L. Brown - 2011

3 papers in library cite

[33]Harmonising Chorales by Probabilistic Inference

M. Allan, Christopher K. I. Williams - 2005

2 papers in library cite

[34]Heterogeneous Acoustic Measurements and Multiple Classifiers for Speech Recognition

A. K. Halberstadt - 1998

2 papers in library cite

[35]An Efficient Approach for Assessing Hyperparameter Importance

Frank Hutter, H. H. Hoos, K. L. Brown - 2014

1 paper in library cites

Felix A. Gers, J. A. P. Ortiz, D. Eck, Jürgen Schmidhuber - 2002

1 paper in library cites

[37]Dictionary of Linguistics and Phonetics

D. Crystal - 2011

1 paper in library cites

[38]Distance Measures for Speech Recognition: Psychological and Instrumental

P. Mermelstein - 1976

1 paper in library cites

[39]Dynamic Cortex Memory: Enhancing Recurrent Neural Networks for Gradient-Based Sequence Learning

S. Otte, M. Liwicki, A. Zell - 2014

1 paper in library cites

[40]Evolving Memory Cell Structures for Sequence Learning

J. Bayer, Daan Wierstra, J. Togelius, Jürgen Schmidhuber - 2009

1 paper in library cites

[41]Fast and Robust Training of Recurrent Neural Networks for Offline Handwriting Recognition

P. Doetsch, M. Kozielski, Hermann Ney - 2014

1 paper in library cites

[42]Generalized Functional ANOVA Diagnostics for High-Dimensional Functions of Dependent Variables

G. Hooker - 2007

1 paper in library cites

[43]Multi-Resolution Linear Prediction Based Features for Audio Onset Detection With Bidirectional LSTM neural networks

E. Marchi, G. Ferroni, F. Eyben, L. Gabrielli, S. Squartini, B. Schuller - 2014

1 paper in library cites

[44]Protein Secondary Structure Prediction With Long Short Term Memory Networks

S. K. Sonderby, O. Winther - 2014

1 paper in library cites

[45]TTS Synthesis With Bidirectional LSTM Based Recurrent Neural Networks

Yu Fan, Y. Qian, F. Xie, F. K. Soong - 2014

1 paper in library cites

Cited by

4

papers in your library

Cites

20

papers in your library

Read

on August 9, 2025

Very good review on different architectural choices of LSTM -  actually brings some nice insights (as opposed to the other paper that compares GRU and LSTMs)

Tags

Paper Aliases

No aliases