manifold mixup

**Manifold Mixup** is an **extension of Mixup that performs interpolation in hidden layer representations rather than the input space** — mixing intermediate features of the network, which creates smoother decision boundaries in the learned representation space. **How Does Manifold Mixup Work?** - **Select Layer**: Randomly choose a hidden layer $k$ from the network. - **Forward**: Pass both input samples to layer $k$ independently. - **Mix**: Interpolate the hidden representations: $ ilde{h}_k = lambda h_k^{(i)} + (1-lambda) h_k^{(j)}$. - **Continue**: Forward the mixed representation through the remaining layers. - **Paper**: Verma et al. (2019). **Why It Matters** - **Better Than Input Mixup**: Mixing in feature space creates more semantically meaningful combinations. - **Flatter Representations**: Produces smoother, more regular hidden representations -> better generalization. - **Multi-Scale**: Randomly selecting the mixing layer provides regularization at multiple abstraction levels. **Manifold Mixup** is **Mixup in thought-space** — blending examples in the network's internal representations for deeper, more meaningful regularization.