maxout

**Maxout** is a learnable activation function that takes the elementwise maximum of several linear transformations. Instead of using a fixed shape such as ReLU or sigmoid, Maxout lets the network learn a piecewise-linear activation directly from data, giving it more flexibility in how it represents nonlinear relationships. **The core idea is simple.** For each input, the network computes several affine candidates and picks the largest one. That creates a convex, piecewise-linear function whose segments are shaped by training. In practice, this can make Maxout useful in hidden layers where richer nonlinearity is helpful, especially in networks that need to model complex decision boundaries. **Why it matters:** Maxout can improve expressiveness and sometimes help models fit difficult functions, but it also costs more memory and compute because each activation unit needs multiple weight sets. It is often discussed as a research-era activation choice that showed the value of learnable nonlinearity, even though ReLU-style activations became more practical in modern systems. | Property | Effect | |---|---| | Piecewise linear | Supports richer nonlinear behavior | | Learnable shape | Adapts to the data during training | | Higher cost | More parameters and compute than ReLU | ```svg ``` In short, Maxout is a powerful but heavier activation design that demonstrates how neural networks can learn their own nonlinearities rather than rely on a fixed activation shape.