Activation functions are the reason depth means anything. Stack a hundred linear layers with no nonlinearity between them and the whole thing collapses algebraically into a single linear map — no amount of depth buys you extra expressive power. The activation is the small element-wise nonlinearity inserted after each layer that breaks this collapse, letting the network bend, fold, and carve the input space into the complex decision regions that deep learning is famous for. Every architectural era has a signature activation, and the migration from ReLU to GELU to gated units like SwiGLU tracks the field's growing understanding of what a good nonlinearity actually needs to do.\n\nReLU — the rectified linear unit — is the workhorse that made very deep networks trainable. It simply passes positive values through and clamps negatives to zero. That gives it a constant gradient of 1 on the positive side, which sidesteps the vanishing-gradient problem that crippled the old saturating activations, and it is almost free to compute. Its one weakness is the dying ReLU problem: a unit stuck in the negative region gets zero gradient forever and stops learning. Leaky ReLU and its cousins patch this by giving the negative side a small nonzero slope so no unit ever fully dies.\n\nThe classic saturating activations — sigmoid and tanh — are now mostly historical. They squash inputs into a bounded range, but their gradients flatten to near-zero for large-magnitude inputs, so gradients vanish through deep stacks. They survive today mainly as gates — inside LSTMs and gated units — where their bounded 0-to-1 output is exactly the "how much to let through" signal you want, rather than as the main activation.\n\nGELU and SiLU/Swish are the smooth successors to ReLU. Instead of a hard kink at zero, GELU weights each input by the probability that a standard Gaussian is below it, producing a smooth curve that dips slightly negative before rising. SiLU (also called Swish) is the closely related x·sigmoid(x). The smoothness gives cleaner gradients and a small but consistent quality gain, which is why GELU became the default inside BERT and the GPT family.\n\nSwiGLU and the gated-linear-unit family are the current default inside large-model feed-forward blocks. A GLU splits the projection into two paths — one carries the signal, the other passes through an activation and gates it by element-wise multiplication. SwiGLU uses a Swish gate, GEGLU uses a GELU gate. Empirically these gated variants outperform a plain activation in the FFN, which is why models like LLaMA and PaLM adopt SwiGLU (usually with a widened hidden size to keep the parameter count matched). The cost is a third weight matrix in the FFN, a trade the quality gain has repeatedly justified.\n\n| Activation | Formula (essence) | Smooth? | Saturates? | Where it lives |\n|---|---|---|---|---|\n| ReLU | max(0, x) | No (kink) | No | CNNs, older nets |\n| Leaky ReLU | x if x>0 else 0.01x | No | No | Fixes dying ReLU |\n| Sigmoid / tanh | squash to bounded range | Yes | Yes | Gates (LSTM/GLU) |\n| GELU / SiLU | x·Φ(x) / x·σ(x) | Yes | No | BERT, GPT blocks |\n| SwiGLU / GEGLU | gated: (act(xW)) ⊙ (xV) | Yes | No | LLM feed-forward |\n\n``svg\n\n``\n\nThe easy way to think about activations is as a menu of curves you pick from by reputation — "use SwiGLU, that's what LLaMA does." The more useful framing is that every activation is answering the same question with a different shape: how should a neuron pass information forward while keeping a usable gradient flowing backward? ReLU's flat-then-linear shape keeps the backward gradient alive; GELU smooths the kink for a cleaner signal; gated units let part of the layer decide how much of the rest to let through. Read an activation through a what-shape-keeps-the-gradient-healthy-and-adds-expressiveness lens rather than a which-curve-is-fashionable lens, and the progression from sigmoid to ReLU to SwiGLU reads as one continuous engineering argument rather than a list of tricks.