lstm

The LSTM — Long Short-Term Memory — is the architecture that fixed the one thing plain recurrent networks could not do: remember across long gaps. A vanilla RNN carries a single hidden state and updates it wholesale at every step, which means its memory is repeatedly overwritten and its training gradients vanish over long sequences. The LSTM's answer is a second, protected memory channel — the *cell state* — that information can ride along almost untouched, edited only through a few carefully learned *gates*. Everything distinctive about the LSTM follows from that one design move: give the network an explicit, gated memory it can choose to keep.\n\n**The cell state is a memory highway, and gates are the on-ramps that edit it.** Running straight through every LSTM step is the cell state, a vector that is only ever modified by simple, mostly-linear operations — a multiply and an add — so a value written into it can survive many steps without decaying. Three gates, each a small sigmoid layer outputting numbers between 0 and 1, control that highway: the *forget gate* decides what fraction of the old memory to erase, the *input gate* decides how much new candidate information to write, and the *output gate* decides how much of the memory to expose as this step's hidden state. Because the gates are learned, the network discovers for itself when to hold a fact and when to drop it.\n\n**That additive memory path is precisely what keeps gradients from vanishing.** In a plain RNN, backpropagation through time repeatedly multiplies by the recurrent weight matrix, and the signal to distant steps shrinks toward zero. The LSTM's cell state instead accumulates through addition, so gradients can flow backward along it almost undiminished — the same "uninterrupted highway" idea that residual connections later brought to very deep feed-forward networks. When the forget gate stays near one, the cell effectively holds a constant, and error signals propagate across hundreds of time steps. This is the concrete mechanism behind the LSTM's ability to model long-range dependencies.\n\n**The GRU is the LSTM stripped to its essentials, and both were eventually outrun by attention.** The Gated Recurrent Unit merges the cell and hidden state and collapses to two gates (update and reset), giving similar accuracy with fewer parameters and slightly faster training; the choice between them is usually empirical. Gated RNNs were the backbone of machine translation, speech recognition, and language modeling from roughly 2014 to 2017. Their limit was never memory but throughput: like all recurrences they must process a sequence step by step and cannot be parallelized across its length, so the transformer — modeling all positions at once with attention — displaced them for large-scale work while the core gating insight lived on.\n\n| Gate | Question it answers | Output |\n|---|---|---|\n| Forget gate | How much old memory to erase? | 0 (drop) to 1 (keep) |\n| Input gate | How much new info to write? | 0 (ignore) to 1 (store) |\n| Output gate | How much memory to reveal now? | 0 (hide) to 1 (expose) |\n| GRU (update + reset) | LSTM's gates, merged and simplified | fewer params |\n\n```svg\n\n```\n\nThe wrong way to learn the LSTM is to memorize the three gate equations and their sigmoids as an arbitrary formula. The useful way is to see that every piece exists to protect one thing: a memory channel that survives across time. The cell state is the memory, the forget and input gates are how the network edits it, the output gate is how it reads from it, and the additive update is the trick that keeps the whole thing trainable over long spans. Read an LSTM through a protect-a-memory-channel-with-learned-gates lens rather than a pile-of-equations lens, and the GRU becomes an obvious simplification, the vanishing-gradient fix becomes inevitable, and even the transformer's later victory reads clearly — attention kept the long-range memory the LSTM won and threw away the sequential bottleneck the LSTM could never escape.