swiglu activation
**SwiGLU and GeGLU Activations** are **gated linear unit (GLU) variants that combine element-wise gating with smooth nonlinearities (Swish or GELU)**, achieving consistent improvements in transformer feedforward network (FFN) quality over standard ReLU or GELU activations — widely adopted in modern large language models including LLaMA, PaLM, and Mistral.
The standard transformer FFN applies: FFN(x) = W2 · activation(W1 · x + b1) + b2, using a single activation function. GLU variants split the first projection into two parallel linear transformations and use one as a gate for the other.
**GLU Family Formulations**:
| Variant | Formula | Activation |
|---------|---------|------------|
| **GLU** | (W1·x) ⊗ σ(V·x) | Sigmoid gate |
| **ReGLU** | (W1·x) ⊗ ReLU(V·x) | ReLU gate |
| **GeGLU** | (W1·x) ⊗ GELU(V·x) | GELU gate |
| **SwiGLU** | (W1·x) ⊗ Swish_β(V·x) | Swish gate |
Here ⊗ denotes element-wise multiplication, W1 and V are separate weight matrices, and Swish_β(x) = x · σ(βx) where σ is the sigmoid function.
**Why Gating Helps**: The gating mechanism allows the network to learn which features to pass through and which to suppress, creating a more expressive transformation than applying a fixed nonlinearity. The multiplicative interaction between the two branches enables the network to learn conditional feature selection — effectively a soft attention mechanism within the FFN.
**Parameter Budget Consideration**: GLU variants use three weight matrices (W1, V, W2) instead of two (W1, W2), increasing FFN parameters by ~50% for the same hidden dimension. To maintain the same parameter count, the hidden dimension is typically reduced by a factor of 2/3. Even with this reduction, GLU variants consistently outperform standard activations at equivalent parameter budgets — the improved expressiveness more than compensates for the reduced width.
**SwiGLU in Practice**: PaLM (540B) uses SwiGLU with FFN hidden dimension = 4d × 2/3 ≈ 2.67d (where d is model dimension). LLaMA uses SwiGLU with hidden dimension rounded to the nearest multiple of 256 for hardware efficiency. The Swish parameter β is typically fixed at 1.0 (reducing to SiLU — Sigmoid Linear Unit).
**Training Stability**: SwiGLU and GeGLU provide smoother gradients than ReLU-based variants (no dead neurons) and avoid the sharp transitions of sigmoid-gated GLU. The smooth gating function helps with gradient flow during training, particularly important for very deep transformer models with hundreds of layers.
**Computational Overhead**: The extra matrix multiplication in GLU variants increases FLOPs by ~50% in the FFN (partially offset by the reduced hidden dimension). On modern GPUs with efficient GEMM implementations, this overhead is minimal — the FFN computation is already compute-bound and well-optimized.
**SwiGLU and GeGLU have become the de facto standard FFN activation for modern LLMs — a simple architectural change that consistently delivers measurable quality gains at negligible additional cost, demonstrating that fundamental activation function choices still matter in the era of scaling.**