gelu
**GELU (Gaussian Error Linear Unit) and SwiGLU** are **activation functions that outperform ReLU in transformer architectures through smooth, probabilistic gating mechanisms** — where GELU gates inputs by their magnitude using the Gaussian CDF (used in BERT, GPT, ViT) and SwiGLU combines Swish activation with a gated linear unit for superior training dynamics (used in LLaMA, PaLM, Gemma), with SwiGLU becoming the standard activation in modern large language models due to consistent empirical accuracy gains.
**What Are GELU and SwiGLU?**
- **GELU**: Defined as x·Φ(x), where Φ is the Gaussian cumulative distribution function — smoothly gates each input by the probability that it would be positive under a standard normal distribution. Unlike ReLU (which hard-clips negatives to zero), GELU provides a smooth, non-monotonic transition that allows small negative values to pass through with reduced magnitude.
- **GELU Approximation**: The exact Gaussian CDF is expensive to compute — the standard approximation is 0.5x(1 + tanh(√(2/π)(x + 0.044715x³))), which is fast and accurate enough for training.
- **SwiGLU**: Defined as Swish(xW₁) ⊙ (xV), combining the Swish activation function (x·σ(βx), where σ is sigmoid) with a Gated Linear Unit (GLU) that uses element-wise multiplication of two linear projections — the gating mechanism allows the network to learn which features to pass through.
- **FFN Architecture Change**: SwiGLU requires three weight matrices in the feed-forward network (FFN) instead of the standard two — but the hidden dimension is reduced to compensate, keeping total parameter count similar while improving quality.
**Why These Activations Matter**
- **No Dead Neurons**: ReLU permanently kills neurons that receive negative inputs (gradient = 0) — GELU and Swish provide non-zero gradients for all inputs, preventing the "dying ReLU" problem that can waste model capacity.
- **Smoother Gradients**: The smooth transitions in GELU and SwiGLU produce more stable gradient flow during training — reducing training instability and enabling faster convergence.
- **Empirical Superiority**: Extensive experiments show SwiGLU consistently outperforms ReLU and GELU in LLM training — Google's PaLM paper demonstrated measurable perplexity improvements from switching to SwiGLU.
- **Industry Standard**: SwiGLU is now the default activation in virtually all modern LLMs — LLaMA, Mistral, Gemma, Qwen, and PaLM all use SwiGLU in their FFN layers.
**Activation Function Comparison**
| Activation | Formula | Properties | Used In |
|-----------|---------|-----------|--------|
| ReLU | max(0, x) | Simple, sparse, dead neurons | Legacy CNNs |
| GELU | x·Φ(x) | Smooth, probabilistic gating | BERT, GPT-2/3, ViT |
| Swish | x·σ(βx) | Smooth, self-gated | EfficientNet |
| SwiGLU | Swish(xW₁) ⊙ xV | Gated, best empirical performance | LLaMA, PaLM, Gemma |
| GeGLU | GELU(xW₁) ⊙ xV | GELU-gated variant | Some research models |
**GELU and SwiGLU are the activation functions powering modern transformer architectures** — replacing ReLU with smooth, gated mechanisms that eliminate dead neurons, improve gradient flow, and deliver consistent accuracy gains, with SwiGLU established as the standard choice for large language model feed-forward networks.