AI summary

This paper introduces Long Short-Term Memory (LSTM), a novel, efficient gradient-based method for recurrent networks. LSTM uses multiplicative gate units to learn and bridge time intervals in excess of 1000 discrete time steps. Experiments on artificial data show that LSTM leads to more successful runs and learns much faster.

Abstract

Learning to store information over extended time intervals via recurrent backpropagation takes a very long time, mostly due to insufficient decaying error back-flow. We briefly review Hochreiter's analysis of this problem, then address it by introducing a novel, efficient gradient-based method called Long Short-Term Memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete time steps by enforcing constant error flow through constant error carrousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O(1). Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with RTRL, BPTT, Recurrent Cascade-Correlation, Elman nets, and Neural Sequence Chunking, LSTM leads to many more successful runs and learns much faster. LSTM also solves complex artificial long time lag tasks that have never been solved by previous recurrent network algorithms.