lipschitz constrained networks

**Lipschitz Constrained Networks** are **neural networks architecturally designed or trained to have a bounded Lipschitz constant** — ensuring that the network's predictions cannot change faster than a specified rate, providing built-in robustness and stability guarantees. **Methods to Constrain Lipschitz Constant** - **Spectral Normalization**: Divide weight matrices by their spectral norm at each layer. - **Orthogonal Weights**: Constrain weight matrices to be orthogonal ($W^TW = I$) — Lipschitz constant exactly 1. - **GroupSort Activations**: Replace ReLU with GroupSort for tighter Lipschitz bounds. - **Gradient Penalty**: Penalize the gradient norm during training to encourage small Lipschitz constant. **Why It Matters** - **Guaranteed Robustness**: A network with Lipschitz constant $L=1$ cannot be fooled by any perturbation that doesn't genuinely change the input class. - **Certified Radius**: $L$ directly gives a certified robustness radius without expensive verification. - **Stability**: Lipschitz-constrained networks are numerically more stable during training and inference. **Lipschitz Constrained Networks** are **sensitivity-bounded models** — architecturally ensuring that outputs change smoothly and predictably with inputs.