maxout

**Maxout** is a **learnable activation function that takes the element-wise maximum of $k$ linear transformations** — effectively learning a piecewise linear activation function whose shape is determined by training data rather than being hand-designed. **How Does Maxout Work?** - **Formula**: $ ext{Maxout}(x) = max_j (W_j x + b_j)$ for $j = 1, ..., k$ (typically $k = 2-5$). - **Piecewise Linear**: The max of $k$ linear functions is a convex piecewise linear function. - **Universal Approximation**: Can approximate any convex function with enough pieces. - **Paper**: Goodfellow et al. (2013). **Why It Matters** - **Learnable Shape**: The activation function's shape is learned from data — not imposed by design. - **Dropout Companion**: Designed to work optimally with dropout regularization. - **Cost**: $k imes$ more parameters and compute than a standard linear layer (one set of weights per piece). **Maxout** is **the activation function that designs itself** — learning the optimal piecewise linear nonlinearity from data.