Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from bioinf.jku.at (Neural Computation 9(8), 1997). This is an open copy from the authors' institute. The official version is in Neural Computation, published by MIT Press.
What it changed
It gave recurrent networks a memory that can keep information for more than 1,000 time steps, by protecting a stored value inside a “memory cell” with learned gates.
The problem before it
A recurrent network reads a sequence one step at a time. It keeps a hidden state, which is a list of numbers that carries information from earlier steps. The network learns by sending an error signal backwards through the steps. This error signal is the gradient: it tells each weight how to change.
Hochreiter had shown in 1991 that this backward signal is multiplied again at every step. So over many steps it either grows without limit or shrinks to almost nothing. When it shrinks, the network cannot learn that an input from long ago caused an output now. In the paper’s own tests, the two standard training methods already failed when 10 steps separated a cue and the point where it mattered.
The idea
The mechanism. The paper builds a special unit called a memory cell. At its centre is one value, the cell state. At each step, the cell keeps its old value and adds the new input that the input gate lets in. The old value is carried over through a self-connection with a fixed weight of 1.0. Because this connection never shrinks or grows the value, the backward error signal also passes along it unchanged. The authors call this path the “constant error carrousel” (their spelling).
A path that never changes is not useful by itself. The cell must decide what to store and when to show it. So the paper adds two gates. A gate is a small unit whose output is between 0 and 1, and the cell multiplies a signal by it:
- The input gate decides when a new input may change the stored value. It protects the memory from inputs that do not matter.
- The output gate decides when the stored value may affect the rest of the network. It protects other units from a memory that is not relevant yet.
Both gates learn their behaviour from data, with the rest of the network. To keep the error flow constant, the training cuts the backward signal at the cell’s inputs. Inside the cell, the signal is never cut.
The notation. For a cell c, the state s_c changes at each step t by s_c(t) = s_c(t-1) + y_in(t) * g(net_c(t)). Here y_in is the input gate’s output, and g squashes the new input. The cell’s output is y_c(t) = y_out(t) * h(s_c(t)), where y_out is the output gate and h squashes the state. The cost per weight and per step is constant, written O(1).
What it showed
Reported in the paper. It was published in a peer-reviewed journal. All tests use artificial tasks that the authors designed to need long memory.
- Embedded Reber grammar. This task predicts the next symbol of strings from a small grammar. Unlike the other methods tested, LSTM learned the task in every run. It also learned faster, even when the other methods’ failed runs are left out. The results table reports means over 30 trials. The other methods were RTRL, Elman networks, recurrent cascade-correlation and neural sequence chunking.
- Long time lags. In tasks with a minimal time lag of 1,000 steps between a cue and the point where it matters, LSTM learned the task. The two standard methods, BPTT and RTRL, already failed when the lag was only 10 steps.
- Noise and continuous values. In further tasks, LSTM learned with noisy inputs and with real-valued numbers. Examples are adding two marked numbers in a long sequence, and telling the order of two widely separated symbols.
Limits, and what came later
The authors list these limits:
- The fast, truncated training cannot solve tasks that give no partial reward for a first step. Their example is the XOR of two inputs far apart in a noisy sequence.
- Like other gradient methods, it cannot count steps exactly. It cannot reliably tell whether a signal came 99 or 100 steps ago.
- Every test used artificial data. Real tasks such as speech and music are named as future work.
The original cell has no way to clear its own memory. A “forget gate” was added in later work (Gers, Schmidhuber and Cummins, 2000), and it is part of the LSTM used today.
LSTM became the standard recurrent unit for about 20 years. Seq2seq is built from stacked LSTMs. The GRU is a simpler gated unit in the same family. CoVe and ELMo take their word representations from LSTMs. The sparsely gated mixture-of-experts layer was first placed between stacked LSTM layers. The Transformer later replaced recurrence in most language models.
What to know first
- How a neural network learns with gradient descent and backpropagation.
- What a recurrent network is: the idea of a hidden state passed from step to step. See Recurrent networks.
- Why repeated multiplication makes numbers explode or vanish. Try multiplying 0.9 by itself 100 times.
How to read it
The paper is 32 pages, and long parts are dense mathematics.
- Read the abstract and section 1 for the problem.
- Read section 3.2 (“Constant error flow: naive approach”). It explains why a plain self-connection is not enough. This is the key step.
- Read section 4 with Figure 1, the memory cell. This is the idea.
- Skim section 5. Read the introduction of each experiment and its results table.
- Read section 6 for the authors’ own list of limits and advantages.
Skip the appendix and the long derivation in section 3.1 on a first reading. Expect about 2 hours for the parts above (an estimate).
Links