mish

**Mish** is a **smooth, self-regularizing activation function defined as $f(x) = x cdot anh( ext{softplus}(x))$** — combining the benefits of Swish-like self-gating with a bounded below property that provides implicit regularization. **Properties of Mish** - **Formula**: $ ext{Mish}(x) = x cdot anh(ln(1 + e^x))$ - **Smooth**: Infinitely differentiable everywhere. - **Non-Monotonic**: Like Swish, has a slight negative region, allowing negative gradients. - **Self-Regularizing**: The bounded-below property prevents activations from going too negative. - **Paper**: Misra (2019). **Why It Matters** - **YOLOv4**: Default activation in YOLOv4 and YOLOv5, where it outperforms Swish and ReLU. - **Marginally Better**: Often 0.1-0.3% better than Swish in practice, though results are architecture-dependent. - **Compute**: Slightly more expensive than Swish due to the tanh(softplus()) composition. **Mish** is **the smooth, self-regulating activation** — a carefully crafted nonlinearity that provides consistent marginal improvements in deep networks.