Adversarial Training (AT) is the most effective defense against adversarial attacks — training the model on adversarial examples by solving a min-max optimization: the inner maximization finds the worst-case perturbation, and the outer minimization trains the model to correctly classify it.

AT Formulation

• Min-Max: $min_ heta mathbb{E}_{(x,y)} [max_{|delta| leq epsilon} L(f_ heta(x+delta), y)]$.
• Inner Max: Use PGD (Projected Gradient Descent) to find the worst-case perturbation $delta^*$.
• Outer Min: Update model parameters to minimize the loss on the perturbed input $x + delta^*$.
• Epsilon: The perturbation budget $epsilon$ defines the robustness guarantee.

Why It Matters

• Gold Standard: AT remains the most reliable defense against adversarial attacks after years of research.
• PGD-AT: Madry et al. (2018) showed that PGD adversarial training provides strong empirical robustness.
• Cost: AT is ~3-10× more expensive than standard training (requires PGD attack at each training step).

Adversarial Training is training on the hardest examples — building robustness by training the model to correctly classify worst-case adversarial perturbations.