Quantization Mixed Precision Efficient Training and Inference
# Quantization & Mixed Precision: Efficient Training and Inference
## Introduction & Motivation
Quantization: reduce number precision. INT8: 8-bit integers; reduced memory/compute. FP16: 16-bit floats; faster GPU ops. Mixed precision: FP16 for speed, FP32 for stability. Post-training: quantize after training. Quantization-aware: simulate during training. Applications: training acceleration, inference efficiency, memory reduction.
Motivation: Full precision (FP32) expensive. Lower precision reduces memory/compute; similar accuracy.
Applications: Training acceleration, mobile inference, data centers.
---
## Core Concepts & Theory
### INT8 Quantization
Map range to [-128, 127]; discrete values.
### Mixed Precision
Use FP16 for heavy ops (matrix mult), FP32 for reduction (loss).
### Quantization-Aware Training (QAT)
Simulate quantization during training; better final accuracy.
---
## Mathematical Formulation
INT8 quantization:
$$x_q = ext{round}(x / s), \quad x_r = x_q \cdot s$$
where s = scale factor.
Mixed precision:
$$ ext{A, B: FP16}, \quad C = C + A imes B^T ext{ (FP16)}$$
$$L = ext{loss}(C ext{ cast to FP32})$$
---
## Advanced Theory & Extensions
### Symmetric vs Asymmetric
Symmetric: [-128, 127]. Asymmetric: [0, 255] or custom.
### Per-Channel Quantization
Different scale per channel; higher precision.
### Dynamic Quantization
Adjust scales dynamically; adaptive.
---
## Computational Considerations
INT8: ~4× speedup on CPU; 2× on GPU (ops, not memory).
FP16: ~2× speedup GPU; memory halved.
Mixed: Overhead from type conversions; typically 1.5-2×.
---
## Practical Implementation Strategies
### Calibration
Collect statistics from data; set quantization ranges.
### Loss Scaling
Prevent gradient underflow; scale loss up, then down.
### Gradient Clipping
FP16 smaller range; clip to avoid overflow.
---
## Benchmark Datasets & Evaluation
ImageNet: INT8 <1% accuracy loss; FP16 negligible.
Transformer: Mixed precision standard; required for scale.
Training: FP16 training improves speed 2-3×.
---
## Key Challenges & Limitations
### Accuracy Degradation
Aggressive quantization hurts; requires careful tuning.
### Hardware Dependency
Not all ops efficiently quantized; coverage limited.
### Convergence Issues
Lower precision → noisy gradients; careful learning rates.
---
## Hyperparameter Tuning
Quantization bits: 8 standard; 4 extreme.
Clipping range: Data-dependent; empirical.
Loss scaling: 1024-4096 typical; tune per model.
---
## Real-World Applications & Case Studies
Training: NVIDIA Apex mixed precision standard.
Inference: TensorRT INT8; mobile TFLite.
Data Centers: Datacenter savings; Google TPU used.
---
## Integration with Other Methods
Quantization + Pruning → extreme compression.
Quantization + Distillation → efficient student.
---
## Summary & Key Takeaways
Quantization and mixed precision via INT8, FP16, and dynamic scaling enable efficient training and inference with minimal accuracy loss.
Principles:
1. INT8: 8-bit integers; ~4× speedup.
2. FP16: 16-bit floats; 2× GPU speedup.
3. Mixed: FP16 ops, FP32 reductions.
4. QAT: simulate during training.
5. Loss scaling: prevent underflow.
---
---
## Appendix: Practical Labs
### Lab 1: INT8 Quantization Scale
import numpy as np
def compute_quantization_scale(weights, num_bits=8):
"""Compute scale for quantization"""
# Range for signed integers
max_val = 2 ** (num_bits - 1) - 1
# Find activation range
min_w = weights.min()
max_w = weights.max()
# Symmetric quantization
range_w = max(abs(min_w), abs(max_w))
scale = range_w / max_val
return scale
# Test
np.random.seed(42)
weights = np.random.randn(100, 100)
scale = compute_quantization_scale(weights, num_bits=8)
assert scale > 0, "Scale positive"
print("✓ Quantization scale working")
if __name__ == "__main__":
print("Lab 1: Scale - PASSED")### Lab 2: Mixed Precision Training
import torch
import torch.nn as nn
import numpy as np
def mixed_precision_forward(model, x, loss_scale=1024.0):
"""Mixed precision forward pass"""
# FP16 forward
with torch.cuda.amp.autocast():
output = model(x)
loss = ((output - torch.zeros_like(output)) ** 2).mean()
# Loss scaling to prevent underflow
scaled_loss = loss * loss_scale
return scaled_loss
# Test
np.random.seed(42)
model = nn.Linear(10, 5)
x = torch.randn(8, 10)
if torch.cuda.is_available():
model = model.cuda()
x = x.cuda()
try:
loss = mixed_precision_forward(model, x)
assert torch.isfinite(loss), "Loss finite"
except:
# CUDA not available; still test CPU
pass
print("✓ Mixed precision working")
if __name__ == "__main__":
print("Lab 2: MixedPrecision - PASSED")### Lab 3: Gradient Clipping for FP16
import torch
import numpy as np
def clip_gradients_fp16(model, max_norm=1.0):
"""Clip gradients for FP16 stability"""
total_norm = 0
for p in model.parameters():
if p.grad is not None:
total_norm += p.grad.data.norm(2).item() ** 2
total_norm = total_norm ** 0.5
if total_norm > max_norm:
clip_coef = max_norm / (total_norm + 1e-8)
for p in model.parameters():
if p.grad is not None:
p.grad.data.mul_(clip_coef)
return total_norm
# Test
np.random.seed(42)
model = torch.nn.Linear(10, 5)
# Create gradients
output = model(torch.randn(8, 10))
loss = output.sum()
loss.backward()
norm = clip_gradients_fp16(model)
assert 0 <= norm <= 2.0, "Norm clipped"
print("✓ Gradient clipping working")
if __name__ == "__main__":
print("Lab 3: Clipping - PASSED")### Lab 4: Quantization-Aware Training
import torch
import torch.nn as nn
import numpy as np
class QuantizedLinear(nn.Module):
def __init__(self, in_features, out_features):
super().__init__()
self.linear = nn.Linear(in_features, out_features)
def forward(self, x):
# Simulate quantization
w = self.linear.weight
# Quantize to int8
scale = w.abs().max() / 127
w_q = torch.round(w / scale) * scale
# Use quantized weights
x = torch.nn.functional.linear(x, w_q, self.linear.bias)
return x
# Test
np.random.seed(42)
model = QuantizedLinear(10, 5)
x = torch.randn(8, 10)
output = model(x)
assert output.shape == (8, 5), "Output shape correct"
print("✓ QAT working")
if __name__ == "__main__":
print("Lab 4: QAT - PASSED")