Mixed Precision Training
# Mixed Precision Training
## Introduction & Motivation
Mixed precision: use FP16 for computation, FP32 for gradients. Faster training, lower memory. Applications: large model training, efficiency.
Motivation: Accelerate training while maintaining numerical stability.
Applications: Large-scale model training, resource-efficient learning.
---
## Core Concepts & Theory
### FP16 Computation
Half-precision forward pass.
### FP32 Master Weights
Full precision parameter updates.
### Loss Scaling
Prevent gradient underflow.
### Gradient Accumulation
Stability with mixed precision.
---
## Mathematical Formulation
Loss Scaling:
$$\mathcal{L}_{ ext{scaled}} = ext{scale} imes \mathcal{L}$$
Unscaling:
$$
abla heta = \frac{
abla \mathcal{L}_{ ext{scaled}}}{ ext{scale}}$$
Speedup Factor:
$$ ext{speedup} \approx 2-3 imes$$
---
## Advanced Theory & Extensions
### Automatic Loss Scaling
Adaptive scaling.
### Gradient Checkpointing
Reduce memory further.
### Dynamic Loss Scaling
Adjust per iteration.
---
## Computational Considerations
Memory: 50% reduction.
Speed: 2-3× faster.
Stability: Careful tuning needed.
---
## Practical Implementation Strategies
### Initial Loss Scale
Choose 1024 or 2048.
### Overflow Handling
Skip updates with overflow.
### Convergence Monitoring
Track loss curves.
---
## Benchmark Datasets & Evaluation
ImageNet: Vision training.
BERT: NLP pretraining.
Large LLMs: Scaling experiments.
---
## Key Challenges & Limitations
### Stability
Careful hyperparameter tuning.
### Overflow
Gradients may overflow.
### Hardware Dependency
Limited hardware support.
---
## Hyperparameter Tuning
Initial scale: 1024-65536.
Scale update: 2x on success.
Max scale: 2^24.
---
## Real-World Applications & Case Studies
Large Models: Train billion-parameter models.
Accelerated Training: 2-3× speedup.
Cost Reduction: Lower compute resources.
---
## Integration with Other Methods
Mixed precision + gradient accumulation; + checkpoint.
---
## Summary & Key Takeaways
Mixed precision enables efficient large-scale training.
Principles:
1. FP16: Fast computation.
2. FP32: Stable updates.
3. Loss scaling: Prevent underflow.
4. Dynamic: Adapt per iteration.
5. Efficiency: 2-3× speedup.
---
## Appendix: Practical Labs
### Lab 1: Loss Scaling
import numpy as np
def scale_loss(loss, scale_factor=1024.0):
"""Scale loss for mixed precision"""
scaled = loss * scale_factor
return scaled
def unscale_gradients(grads, scale_factor=1024.0):
"""Unscale gradients after backward"""
unscaled = grads / scale_factor
return unscaled
loss = 2.5
scaled = scale_loss(loss, 1024)
assert scaled > loss
grads = np.array([1e-3, 2e-3])
unscaled = unscale_gradients(grads * scaled, 1024)
print("✓ Loss scaling working")### Lab 2: Overflow Handling
import numpy as np
def detect_overflow(gradients, max_val=1e4):
"""Detect gradient overflow"""
overflow = np.any(np.abs(gradients) > max_val) or np.any(np.isnan(gradients))
return overflow
np.random.seed(42)
grads = np.random.randn(100)
overflow = detect_overflow(grads)
assert not overflow
grads[0] = np.inf
overflow = detect_overflow(grads)
assert overflow
print("✓ Overflow detection working")### Lab 3: Dynamic Loss Scaling
class DynamicLossScaler:
def __init__(self, init_scale=1024, scale_factor=2):
self.scale = init_scale
self.scale_factor = scale_factor
self.overflow_count = 0
def scale_loss(self, loss):
return loss * self.scale
def update_scale(self, overflow):
if overflow:
self.scale = self.scale / self.scale_factor
self.overflow_count += 1
else:
self.scale = self.scale * self.scale_factor
scaler = DynamicLossScaler()
scaler.update_scale(False)
assert scaler.scale > 1024
print("✓ Dynamic scaler working")### Lab 4: Memory Savings
def estimate_memory_savings(model_size, gradient_size):
"""Estimate memory savings from mixed precision"""
# FP32: 4 bytes, FP16: 2 bytes
fp32_memory = model_size * 4 + gradient_size * 4
mixed_memory = model_size * 4 + gradient_size * 2
savings = (fp32_memory - mixed_memory) / fp32_memory
return savings
savings = estimate_memory_savings(1e9, 1e9)
assert 0 < savings < 0.5
print(f"✓ Memory savings: {savings:.1%}")---