glu

**GLU** (Gated Linear Unit) is a **gating mechanism that splits the input into two halves — one serves as the "content" and the other as the "gate"** — implemented as $ ext{GLU}(x, y) = x otimes sigma(y)$ where $otimes$ is element-wise multiplication. **How Does GLU Work?** - **Split**: Given input of dimension $2d$, split into $x$ and $y$ of dimension $d$ each. - **Gate**: $ ext{GLU}(x, y) = x otimes sigma(y)$ - **Variants**: Bilinear ($x otimes y$), SwiGLU ($x otimes ext{Swish}(y)$), GeGLU ($x otimes ext{GELU}(y)$). - **Paper**: Dauphin et al. (2017). **Why It Matters** - **LLM Standard**: SwiGLU/GeGLU variants are the default FFN activation in modern LLMs (LLaMA, PaLM, Gemma). - **Gradient Flow**: The linear path through $x$ provides easy gradient flow (like a skip connection within the activation). - **Performance**: GLU variants consistently outperform standard ReLU/GELU FFN blocks in transformers. **GLU** is **the half-and-half activation** — splitting inputs into content and gate for multiplicative feature selection.