self-gating

**Self-Gating** is a **mechanism where a neural network layer gates its own activations using a function of the same input** — the input multiplied by a sigmoid (or similar gate) of itself, allowing the network to selectively amplify or suppress its features. **How Does Self-Gating Work?** - **Formula**: $y = x cdot sigma(Wx + b)$ where $sigma$ is a gate function (sigmoid, tanh). - **Swish**: The simplest self-gating: $x cdot sigma(x)$ (no learned gate parameters). - **SE-Net**: Self-gating via channel attention: learn per-channel gates from global statistics. - **GLU**: Gated Linear Unit splits input into two halves — one gates the other. **Why It Matters** - **Expressiveness**: Self-gating allows multiplicative interactions, which are more expressive than additive transformations. - **Feature Selection**: The gate learns to suppress irrelevant features and amplify important ones. - **Foundation**: Self-gating is the core principle behind Swish, GLU, SwiGLU, and SE-Net. **Self-Gating** is **the input controlling its own flow** — a powerful mechanism where features decide their own importance.