SwiGLU gated linear units

**SwiGLU and Gated Linear Units in Transformers** are **advanced activation architectures where feed-forward networks use gated mechanisms to selectively combine multiple transformation branches — achieving higher capacity per parameter than ReLU networks with 30% parameter reduction for equivalent performance**. **Gated Linear Unit (GLU) Fundamentals:** - **Gate Mechanism**: splitting dimension D into two branches: y = (W₁x ⊙ σ(W₂x)) where ⊙ is element-wise multiplication and σ is sigmoid function - **Gating Effect**: sigmoid output σ(W₂x) ∈ [0,1] acts as soft gate selecting which dimensions from W₁x to pass — learned dynamic routing - **Parameter Efficiency**: maintaining output dimension D while using 2D input projection (2×D parameters) vs traditional expansion 4D - **Variant Forms**: variants include Bilinear (y = W₁x ⊙ W₂x), Tanh-gated (y = W₁x ⊙ tanh(W₂x)), and linear gated architectures **SwiGLU Architecture:** - **Swish Activation**: replacing standard sigmoid gate with Swish (SiLU): y = (W₁x) ⊙ SiLU(W₂x) where SiLU(z) = z·sigmoid(z) - **Gating Function**: SiLU provides smoother gradient flow compared to sigmoid — 0.5-1.0 at zero, approaching linear for large values - **Capacity Enhancement**: SwiGLU with intermediate dimension 2.67D achieves same performance as ReLU with 4D — 33% parameter reduction - **Empirical Validation**: PaLM models using SwiGLU consistently outperform ReLU baseline by 1-2% accuracy across downstream tasks **Transformer Feed-Forward Integration:** - **Traditional FFN**: two linear layers with ReLU: FFN(x) = ReLU(W₁x)W₂ with output dimension d_model, intermediate 4×d_model - **GLU Variant FFN**: GLU(x) = (W₁x ⊙ σ(W₂x))W₃ with 3 linear layers, intermediate typically 2.67×d_model or 8/3×d_model - **Parameter Count**: SwiGLU(d) ≈ 2.67 × d × d vs traditional FFN 4 × d × d — 33% reduction while maintaining or improving performance - **Computation**: SwiGLU requires 3 matrix multiplications vs 2 for ReLU — ~1.5x compute per token despite parameter reduction **Performance Benchmarks:** - **PaLM Models**: 8B PaLM with SwiGLU matches 10B with ReLU on downstream tasks (SuperGLUE 90.2% vs 89.8%) — clear parameter efficiency - **Scaling Laws**: SwiGLU-based models scale more efficiently with data, requiring 10-15% fewer training tokens for target performance - **Fine-tuning**: SwiGLU-based models fine-tune more effectively on low-data tasks — 3-5% improvement on few-shot classification - **Downstream Transfer**: consistent 1-2% improvements across MMLU, HellaSwag, TruthfulQA — holds across model scales 8B to 540B **Mathematical Properties:** - **Gradient Flow**: SwiGLU gradient ∂y/∂x includes both multiplicative (gate) and additive (Swish) components — richer gradient signal than ReLU - **Non-linearity**: SwiGLU introduces stronger non-linearity (second-order polynomial in gate component) vs ReLU (piecewise linear) - **Activation Saturation**: gate output σ(x) saturates to 0 or 1 for extreme inputs, providing regularization effect — reduces need for explicit dropout - **Inductive Bias**: gating mechanism biases toward sparse activation patterns (some dimensions suppressed per-token) — aligns with lottery ticket hypothesis **Comparative Activation Functions:** - **ReLU**: simple, linear for positive inputs, zero for negative — foundation of deep learning but gradient-starved in sparse settings - **GELU**: smooth approximation of ReLU with element-wise probability gate — better gradient flow, used in BERT and GPT-2 - **SiLU (Swish)**: self-gated activation x·sigmoid(x), smooth everywhere — improves over ReLU by 1-2% in language models - **GLU Variants**: bilinear, tanh-gated, linear-gated all provide gating benefits — SwiGLU empirically optimal for transformers **Implementation Details:** - **Llama Models**: recent Llama versions use SwiGLU gate activation with 2.67× intermediate dimension — standard for frontier models - **PaLM Architecture**: introduced SwiGLU and demonstrated consistent improvements across all parameter scales — influential for modern designs - **Inference Optimization**: gating provides implicit sparsity (30-40% of neurons inactive per token) — enables 20-30% speedup with structured pruning - **Scaling Consideration**: SwiGLU adds 50% computation per token compared to ReLU-based 4D intermediate — balanced by parameter efficiency **SwiGLU and Gated Linear Units in Transformers represent modern activation design — enabling more parameter-efficient models with improved performance through learned gating mechanisms that rival or exceed traditional feed-forward networks.**