AI summary

This paper introduces a forget gate mechanism for LSTM networks to address the issue of unbounded growth in cell states when processing continual input streams, achieving improved performance on a continual version of the embedded Reber grammar benchmark.

Abstract

Long Short-Term Memory (LSTM, [5]) can solve many tasks not solvable by previous learning algorithms for recurrent neural networks (RNNs). We identify a weakness of LSTM networks processing continual input streams without explicitly marked sequence ends. Without resets, the internal state values may grow indefinitely and eventually cause the network to break down. Our remedy is an adaptive "forget gate" that enables an LSTM cell to learn to reset itself at appropriate times, thus releasing internal resources. We review an illustrative benchmark problem on which standard LSTM outperforms other RNN algorithms. All algorithms (including LSTM) fail to solve a continual version of that problem. LSTM with forget gates, however, easily solves it in an elegant way.