adversarial robustness adversarial examples attacks defenses and verification
# Adversarial Robustness: Adversarial Examples, Attacks, Defenses, and Verification
## 1. Introduction & Motivation
Adversarial examples are carefully crafted inputs designed to fool machine learning models. A slightly perturbed image might be imperceptibly different to humans yet dramatically change model predictions. This vulnerability is fundamental to deep learning and poses serious security risks:
- Safety-critical systems: Autonomous vehicles misidentifying stop signs
- Security systems: Face recognition fooled by adversarial patches
- Content moderation: Adversarial examples bypass detection
Adversarial robustness research aims to develop models resistant to such attacks. This article covers adversarial attack methods, defense mechanisms, certified robustness, and practical deployment considerations.
## 2. Core Concepts & Theory
### 2.1 Adversarial Examples
Formally, for sample x with label y, an adversarial example x' satisfies:
$$ ext{Adversarial}(x') \iff \begin{cases} f(x') eq y \\ \|x - x'\|_p \leq \epsilon \quad ext{(small perturbation)} \end{cases}$$
where
$$ \epsilon $$
is maximum allowed perturbation (e.g., 8/255 for images).
Threat models:
- Evasion: Attacker modifies input at test time
- Poisoning: Attacker contaminates training data
- Backdoor: Attacker injects hidden triggers
This article focuses on evasion attacks.
### 2.2 Fast Gradient Sign Method (FGSM)
Simple, efficient attack using single gradient step:
$$x' = x + \epsilon \cdot ext{sign}( abla_x \mathcal{L}(x, y))$$
Attack in direction of increasing loss. Extremely fast (~1ms per image) but often suboptimal.
### 2.3 Projected Gradient Descent (PGD)
Iterative attack, stronger than FGSM:
$$x_{t+1} = ext{Clip}(x_t + \alpha \cdot ext{sign}( abla_x \mathcal{L}(x_t, y)))$$
where Clip enforces
$$ \|x_t - x\|_\infty \leq \epsilon $$
. Stronger: Uses multiple steps to find better adversarial.
Cost: O(T) forward-backward passes for T iterations (typically 20-100).
### 2.4 Adversarial Training
Robust model trained on adversarial examples:
$$\min_ heta \mathbb{E}_{x,y}[\max_{\|x' - x\| \leq \epsilon} \mathcal{L}(x', y; heta)]$$
Inner maximization finds worst-case perturbation; outer minimization updates parameters. Results in more robust but slightly less accurate model.
## 3. Mathematical Formulation
### 3.1 Different Threat Models
L∞ norm (Linf): Maximum element-wise change
$$\|x - x'\|_\infty = \max_i |x_i - x'_i| \leq \epsilon$$
Common:
$$ \epsilon = 8/255 \approx 0.03 $$
L2 norm: Euclidean distance
$$\|x - x'\|_2 = \sqrt{\sum_i (x_i - x'_i)^2} \leq \epsilon$$
Less common; harder for defenses
L0 norm: Number of changed pixels
$$\|x - x'\|_0 = |\{i : x_i
eq x'_i\}| \leq k$$
Natural for sparse perturbations; most adversarial research uses L∞
### 3.2 Robust Optimization Objective
Standard training:
$$\min_ heta \mathbb{E}[\mathcal{L}(f_ heta(x), y)]$$
Robust training:
$$\min_ heta \mathbb{E}[\max_{\|x'-x\| \leq \epsilon} \mathcal{L}(f_ heta(x'), y)]$$
Inner max is attack; outer min is defense. Saddle point problem, challenging to optimize.
### 3.3 Certified Robustness via Randomized Smoothing
Smooth model by adding Gaussian noise:
$$f_c(x) = \arg\max_c \mathbb{P}_{u \sim \mathcal{N}(0, \sigma^2 I)}[f(x + u) = c]$$
Certification: For correctly classified x, certified robustness radius:
$$R = \frac{\sigma}{2}[\Phi^{-1}(p_A) - \Phi^{-1}(p_B)]$$
where
$$ p_A $$
is probability of top class,
$$ p_B $$
second class,
$$ \Phi $$
is normal CDF.
### 3.4 Robustness Verification via Abstraction
Interval bound propagation: Track feasible input range through network.
For layer
$$ i+1 $$
:
$$[\ell_{i+1}, u_{i+1}] = ext{ReLU}(W[\ell_i, u_i] + b)$$
Efficiently verify that for any input in
$$ [\ell_0, u_0] $$
, output constraint satisfied.
## 4. Advanced Theory & Extensions
### 4.1 Certified Defenses with Randomized Smoothing
Strength: Provides formal robustness guarantees (vs. heuristic defenses).
Process:
1. Train base model f with data augmentation (noise injection)
2. Certify: Compute guaranteed robustness radius
3. Larger
$$ \sigma $$
(training noise) → larger certified radius
Typical:
$$ \epsilon = 0.5 $$
certified robustness with modest accuracy loss.
### 4.2 Provable Defenses via Abstract Interpretation
Abstractly interpret neural networks to verify properties.
Interval arithmetic: Track bounds on activations
- Forward pass:
$$ ([\ell_i, u_i], W, b) o [\ell_{i+1}, u_{i+1}] $$
- Constraint: Check all possible outputs satisfy property
Tighter bounds → tighter verification → less conservative.
### 4.3 Certified Defenses via Convex Relaxations
Formulate robustness verification as optimization:
$$ ext{Robust if} \quad \min_{\|x'-x\| \leq \epsilon} f(x') \geq 0$$
Convex relaxation (semidefinite programming, linear programming) computes bounds.
Trade-off: More compute → tighter bounds → higher certified robustness.
### 4.4 Robust Model Merging
Fine-tune robust base model for specific task:
$$ heta_{ ext{target}} = heta_{ ext{robust}} + \alpha \Delta heta$$
Small
$$ \alpha $$
preserves robustness. Empirically: 50-60% of robust model knowledge transfers.
## 5. Computational Considerations
### 5.1 Attack Cost
FGSM: 1 backward pass = ~3ms (CNN)
PGD-100: 100 backward passes = ~300ms
Adaptive attacks: Specialized for defense, can be 1000x slower
Total evaluation on 10K images:
- FGSM: ~30s
- PGD-100: ~50m
- Adaptive: Several hours
### 5.2 Defense Cost: Adversarial Training
Standard training: 100 epochs ~12 hours
Adversarial training: 500+ epochs (generating adversarial per iteration) ~120 hours (10x overhead)
### 5.3 Certification Cost
Randomized smoothing: O(N) where N is samples for Monte Carlo estimation
- Certification: 1000-10000 forward passes per image
- Impractical for real-time (~1-10s per image)
Abstract interpretation:
$$ O(d^3) $$
where d is dimension
- Feasible for smaller networks
- Intractable for large networks
### 5.4 Inference Overhead
- Standard model: 100ms per image
- Adversarially trained model: 105ms (~5% overhead)
- Smoothed model: 10-100ms per image (10x for certification)
## 6. Practical Implementation Strategies
### 6.1 Adversarial Training Procedure
Repeat for num_epochs:
For batch (x, y):
1. Generate adversarial: x' = Attack(x, model, eps)
2. Forward on adversarial: loss = CrossEntropy(model(x'), y)
3. Update model to minimize lossImplementation details:
- Attack method: PGD-7 typical (balance between strength and speed)
- Epsilon: 8/255 standard for image classification
- Training multiplier: 2-5x longer than standard
### 6.2 Choosing Epsilon (Perturbation Budget)
L∞ perturbation 8/255 ≈ 0.03:
- Visual impact: Imperceptible to humans
- Model impact: Significant (>95% success rate)
- Standard evaluation benchmark
Alternative budgets:
- 4/255: Smaller perturbations, easier to defend
- 16/255: Larger perturbations, harder to defend
### 6.3 Attack Selection for Evaluation
Weak attacks (quick evaluation):
- FGSM: Unreliable (often overestimates robustness)
- PGD-20: ~50 iterations, reasonable estimate
Strong attacks (definitive evaluation):
- PGD-100: 100 iterations
- Adaptive attacks: Account for defense specifics
- Ensemble of attack methods
### 6.4 Defense Composition
Combining multiple defenses often provides minimal additional benefit (diminishing returns). However, ensemble defenses can help:
$$f_{ ext{ensemble}}(x) = ext{argmax}_c \sum_i p_{i,c}(x)$$
Ensemble of 5-10 robust models improves robustness by 5-15%.
## 7. Benchmark Datasets & Evaluation
### 7.1 Adversarial Robustness Benchmarks
RobustBench: Standardized evaluation of robust models
CIFAR-10 (L∞, ε=8/255):
- Standard (no defense): 95% accuracy, 0% robust accuracy
- Adversarially trained: 87% accuracy, 57% robust accuracy
- Certified defenses: 82% accuracy, 45% certified robustness
ImageNet (L∞, ε=4/255):
- Standard: 76% accuracy, 0% robust accuracy
- Adversarially trained: 67% accuracy, 30% robust accuracy
- Much harder than CIFAR-10; robustness still limited
### 7.2 Evaluation Metrics
Robust Accuracy: % correct on adversarially perturbed inputs
$$ ext{RobustAcc} = \frac{|\{x : f(x') = y, \|x'-x\| \leq \epsilon\}|}{N}$$
where x' is strongest adversarial example found.
Certified Robustness: Radius where model guaranteed correct
$$R = \max_{\epsilon'} \{\epsilon' : f(x) = f(x') \forall \|x'-x\| \leq \epsilon'\}$$
Averaged over test set.
### 7.3 Benchmark Results
CIFAR-10 (8/255):
- Adversarial training: 57% robust accuracy (vs 95% clean)
- Certified via smoothing: 45% certified, 65% clean accuracy
- Trade-off: ~10-15% clean accuracy loss for robustness
ImageNet (4/255):
- Adversarial training: 30% robust accuracy (vs 76% clean)
- Certified: 15-20% certified robustness
- Much harder than CIFAR-10
## 8. Key Challenges & Limitations
### 8.1 Robustness-Accuracy Trade-off
Fundamental trade-off: Making model robust to perturbations hurts clean accuracy:
$$ ext{Robust model: } 87\% ext{ robust, } 85\% ext{ clean}$$
$$ ext{Standard model: } 0\% ext{ robust, } 95\% ext{ clean}$$
No known way to eliminate this trade-off. Larger models help but don't eliminate it.
### 8.2 Limited Certified Robustness
Certified methods (randomized smoothing, verification) provide formal guarantees but with large gaps:
- Empirical robustness (via PGD): 60-70%
- Certified robustness: 40-50%
- Gap indicates certified bounds too conservative
### 8.3 Adaptive Attacks
As defenses improve, attackers adapt by changing attack strategy:
- Defense optimizes against specific attack (e.g., FGSM)
- Adaptive attacker designs new attack accounting for defense
- Adaptive attack often breaks defense
Requires careful adaptive attack evaluation for each defense.
### 8.4 Generalization of Adversarial Training
Models trained on ε-perturbations robust to ε but not ε/2 or 2ε:
$$ ext{RobustAcc}(\epsilon/2) = 95\%, \quad ext{RobustAcc}(\epsilon) = 57\%, \quad ext{RobustAcc}(2\epsilon) = 10\%$$
Robustness specific to perturbation budget; limited generalization.
## 9. Hyperparameter Tuning & Optimization
### 9.1 Adversarial Training Parameters
Attack iterations (PGD steps): 7-20 typical
- More iterations: Stronger attacks, slower training
- Diminishing returns beyond 20 steps
- Typical: PGD-7 for fast training, PGD-20 for evaluation
Step size: α = ε/iterations typical
- Balances convergence and exploration
- Smaller α: More careful but slower
- Standard: α = 2.5 · ε / iterations
Epsilon (perturbation budget): Task-dependent
- Image: 8/255 standard
- Smaller ε: Easier defense
- Larger ε: Harder defense
### 9.2 Certified Defense Parameters
Noise level (σ): Controls certification radius
- Larger σ: Larger certified radius, lower clean accuracy
- Typical: σ ∈ [0.12, 1.0]
- Relationship: Certified radius ≈ 0.5σ
Sampling for certification: Trade accuracy for compute
- Fewer samples: Faster, less accurate
- Typical: 100-1000 samples per image for certification
### 9.3 Learning Rate and Optimization
Learning rate: Start high, decay over time
- Initial: 0.01-0.1 (conservative for robust training)
- Decay: Multi-step or cosine
- Lower LR needed than standard training (robust loss landscape harder)
Optimizer: SGD with momentum common
- β1 = 0.9, β2 = 0.999 for Adam
- Warmup: 10-20% of training
## 10. Real-World Applications & Case Studies
### 10.1 Autonomous Vehicle Adversarial Robustness
Problem: Self-driving car perception attacked by adversarial patches
Setup:
- Model: YOLO object detector
- Threat: Stop sign misclassified as speed limit sign
- Scenario: Attacker places adversarial patch on road sign
Adversarial Training Approach:
- Train on CIFAR-10 perturbed by stop sign patch patterns
- Adversarial training with ε = 16/255 (larger for physical world)
- Combine with input preprocessing (defense)
Results:
- Standard model: 95% misclassification under patch attack
- Adversarially trained: 15% misclassification (85% robust)
- Deployment: Added to safety pipeline, works as part of ensemble
Real-world considerations:
- Adversarial patches must survive physical world
- Combined defense needed (not just adversarial training)
- Trade-off: Slightly slower convergence on traffic detection
### 10.2 Medical Image Analysis Robustness
Problem: Radiology AI system must be robust to adversarial inputs
Setup:
- Model: ResNet-50 fine-tuned for pneumonia detection
- Threat: Adversarial perturbations on CT scans
- Requirement: Robust to ε = 4/255 perturbations (very small in medical context)
Adversarial Training:
- PGD-20 attacks during training
- Certified randomized smoothing on top
- ε = 4/255 (conservative, small perturbations)
Results:
- Standard model: 98% accuracy, 0% robust accuracy
- Adversarially trained: 96% accuracy, 68% robust accuracy
- Certified: 94% accuracy, 45% certified robustness
Medical considerations:
- Small ε (4/255) appropriate for expert review
- Human + AI decision-making reduced adversarial risk
- Clinical deployment approved
### 10.3 Content Moderation Spam Detection
Problem: Email spam detector attacked by adversarial emails
Setup:
- Model: Gradient boosting tree classifier
- Threat: Adversarial text modified to evade detection
- Goal: Robustness to character/word substitutions
Defense:
- Feature squashing: Reduce feature precision
- Ensemble: Multiple detectors, vote
- Adversarial training (limited, text models hard to train robustly)
Results:
- Standard: 95% detection rate, easily evaded
- With defenses: 82% detection, 20% harder to evade
- Deployed: Trade accuracy for robustness acceptable
Practical notes:
- Text adversarial examples less studied
- Certified defenses for NLP still nascent
- Heuristic defenses more common
### 10.4 Biometric Authentication (Face Recognition)
Problem: Face recognition fooled by adversarial glasses or makeup
Setup:
- Model: CLIP-based face verification
- Threat: Adversarial eyeglasses patterns
- Constraint: Must work with any lighting, angle
Robust Training:
- Augmentation: Simulate eyeglasses on faces
- Adversarial training: L∞ perturbations on faces
- Physical robustness: Train on images with adversarial patterns physically applied
Results:
- Standard model: 95% accuracy, ~90% fooled by crafted eyeglasses
- Adversarially trained: 92% accuracy, 30% fooled by eyeglasses
- Liveness detection: Added as secondary check
## 11. Integration with Other Methods
### 11.1 Ensemble Defenses
Combine multiple models:
$$p(y|x) = \frac{1}{M} \sum_{i=1}^{M} p_i(y|x)$$
If models trained differently, attacker harder to adapt.
Benefit: 5-15% robustness improvement, modest computation cost.
### 11.2 Input Preprocessing Defenses
Denoise input before classification:
$$x' = ext{Denoise}(x)$$
Can remove adversarial perturbations but also remove legitimate features (accuracy drop).
### 11.3 Detection-Based Defense
Detect adversarial examples rather than classify robustly:
$$f(x) = \begin{cases} ext{Classify}(x) & ext{if not adversarial} \\ ext{Abstain} & ext{if adversarial detected} \end{cases}$$
Practical but not robust (detection can also be attacked).
### 11.4 Certified Defenses + Adversarial Training
Combine certified randomized smoothing with adversarial training:
1. Train base model with adversarial training
2. Apply randomized smoothing for certification
Better certified robustness than smoothing alone on unrobust base.
## 12. Future Research Directions
### 12.1 Certified Robustness at Scale
Current: Limited to small models/networks
Goal: Certify large models (ResNet-50, ViT) on ImageNet
Requires:
- Tighter verification bounds
- Scalable certification algorithms
- Better theoretical understanding
### 12.2 Robustness to Multiple Threat Models
Current: Robustness to single norm (L∞)
Goal: Robust to multiple perturbation types simultaneously
Challenge: Trade-off between defending against different threats.
### 12.3 Efficient Adversarial Training
Current: 5-10x slower than standard training
Goal: Minimal training overhead while maintaining robustness
Recent progress with fast attacks (FGSM+) and model architecture tuning.
### 12.4 Certified Robustness for Complex Tasks
Current: Only image classification studied
Goal: Certification for detection, segmentation, NLP tasks
Much harder; requires rethinking certification approaches.
## 13. Summary & Key Takeaways
Attack Methods:
- FGSM: Fast (1ms) but weak
- PGD-20: Good balance (50ms), standard evaluation
- Adaptive attacks: Strongest but slow (seconds)
Defense Strategies:
- Adversarial training: Practical, 57% robust acc. on CIFAR-10 (vs 0%)
- Certified smoothing: Provable guarantees, 45% certified acc.
- Verification: Formal guarantees, conservative bounds
- Ensemble: 5-15% improvement, modest cost
Robustness-Accuracy Trade-off:
- Standard model: 95% clean accuracy, 0% robust
- Robust model: 85% clean accuracy, 57% robust
- Fundamental trade-off; no way to eliminate
Hyperparameters:
- PGD iterations: 7-20 (more = stronger attacks)
- Perturbation budget: 8/255 standard
- Learning rate: Lower than standard training
- Training cost: 5-10x longer than standard
Performance:
- CIFAR-10 robustness: ~57% vs 95% clean (10% drop)
- ImageNet robustness: ~30% vs 76% clean (46% drop)
- Harder for larger, more complex models
Limitations:
- Limited certified robustness guarantees
- Adaptive attacks can break defenses
- Robustness specific to perturbation budget
- Generalization limited across different epsilons
Adversarial robustness is increasingly important for deployed ML systems. Practical deployments use adversarial training + ensemble defenses; certified robustness still limited but improving.
---
## Appendix: Practical Implementation Labs
### Lab 1: FGSM and PGD Attacks
import torch
import torch.nn as nn
def fgsm_attack(model, x, y, epsilon=8/255):
"""Fast Gradient Sign Method attack"""
x = x.clone().requires_grad_(True)
output = model(x)
loss = nn.CrossEntropyLoss()(output, y)
loss.backward()
# Generate adversarial
x_adv = x + epsilon * x.grad.sign()
x_adv = torch.clamp(x_adv, 0, 1)
return x_adv.detach()
def pgd_attack(model, x, y, epsilon=8/255, alpha=2/255, steps=20):
"""Projected Gradient Descent attack"""
x_adv = x.clone()
for _ in range(steps):
x_adv = x_adv.clone().requires_grad_(True)
output = model(x_adv)
loss = nn.CrossEntropyLoss()(output, y)
loss.backward()
# Gradient step
x_adv = x_adv + alpha * x_adv.grad.sign()
# Projection
x_adv = torch.clamp(x_adv, x - epsilon, x + epsilon)
x_adv = torch.clamp(x_adv, 0, 1)
return x_adv.detach()### Lab 2: Adversarial Training
def adversarial_training_step(model, x, y, optimizer, epsilon=8/255):
"""Single adversarial training step"""
# Generate adversarial examples
x_adv = pgd_attack(model, x, y, epsilon=epsilon, steps=7)
# Train on adversarial examples
optimizer.zero_grad()
output = model(x_adv)
loss = nn.CrossEntropyLoss()(output, y)
loss.backward()
optimizer.step()
return loss.item()
def train_robust_model(model, train_loader, num_epochs, epsilon=8/255):
"""Train adversarially robust model"""
optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
for epoch in range(num_epochs):
for x, y in train_loader:
adversarial_training_step(model, x, y, optimizer, epsilon)
scheduler.step()### Lab 3: Robustness Evaluation
def evaluate_robustness(model, test_loader, epsilon=8/255, num_steps=20):
"""Evaluate model robustness to PGD attacks"""
model.eval()
correct = 0
robust_correct = 0
for x, y in test_loader:
# Clean accuracy
with torch.no_grad():
clean_output = model(x)
correct += (clean_output.argmax(1) == y).sum().item()
# Robust accuracy (PGD attack)
x_adv = pgd_attack(model, x, y, epsilon=epsilon, steps=num_steps)
with torch.no_grad():
adv_output = model(x_adv)
robust_correct += (adv_output.argmax(1) == y).sum().item()
total = len(test_loader.dataset)
print(f"Clean accuracy: {100*correct/total:.1f}%")
print(f"Robust accuracy: {100*robust_correct/total:.1f}%")
return correct / total, robust_correct / total### Lab 4: Certified Robustness via Randomized Smoothing
def randomized_smoothing_predict(model, x, num_samples=1000, sigma=0.5):
"""Predict with randomized smoothing"""
counts = torch.zeros(model.num_classes)
for _ in range(num_samples):
# Add Gaussian noise
noise = torch.randn_like(x) * sigma
x_noisy = torch.clamp(x + noise, 0, 1)
with torch.no_grad():
output = model(x_noisy)
pred = output.argmax(1)
counts[pred] += 1
prediction = counts.argmax()
return prediction, counts
def certify_robustness(pred_counts, sigma, num_samples, delta=0.001):
"""Compute certified robustness radius"""
from scipy.stats import norm
counts_sorted = sorted(pred_counts.numpy())
n_A = counts_sorted[-1] # Most common class
n_B = counts_sorted[-2] # Second most common
p_A = n_A / num_samples
p_B = n_B / num_samples
# Certification radius
phi_inv_A = norm.ppf(p_A - delta / 2)
phi_inv_B = norm.ppf(p_B + delta / 2)
radius = (sigma / 2) * (phi_inv_A - phi_inv_B)
return max(0, radius)