dynamic quantization
**Dynamic quantization** determines quantization parameters (scale and zero-point) **at runtime** based on the actual values flowing through the network during inference, rather than using fixed parameters determined during calibration.
**How It Works**
- **Weights**: Quantized statically (ahead of time) and stored in INT8 format.
- **Activations**: Remain in floating-point during computation. Quantization parameters are computed **dynamically** for each batch based on the observed min/max values.
- **Computation**: Matrix multiplications and other operations are performed in INT8, but activations are quantized on-the-fly.
**Workflow**
1. **Load**: Load pre-quantized INT8 weights.
2. **Observe**: For each activation tensor, compute min/max values from the current batch.
3. **Quantize**: Compute scale and zero-point, quantize activations to INT8.
4. **Compute**: Perform INT8 operations (e.g., matrix multiplication).
5. **Dequantize**: Convert results back to FP32 for the next layer.
**Advantages**
- **No Calibration**: No need for a calibration dataset to determine activation ranges — the model adapts to the actual input distribution at runtime.
- **Accuracy**: Often achieves better accuracy than static quantization because it adapts to each input's specific value range.
- **Easy to Apply**: Can be applied post-training without retraining or fine-tuning.
**Disadvantages**
- **Runtime Overhead**: Computing min/max and quantization parameters for each batch adds latency (typically 10-30% slower than static quantization).
- **Variable Latency**: Inference time varies depending on input value ranges.
- **Limited Speedup**: Activations are quantized/dequantized repeatedly, reducing the efficiency gains compared to static quantization.
**When to Use Dynamic Quantization**
- **Recurrent Models**: LSTMs, GRUs, and Transformers where activation ranges vary significantly across sequences.
- **Variable Input Distributions**: When inputs have unpredictable value ranges (e.g., user-generated content).
- **Quick Deployment**: When you need quantization benefits without the effort of calibration.
**PyTorch Example**
```python
import torch
model = MyModel()
quantized_model = torch.quantization.quantize_dynamic(
model,
{torch.nn.Linear, torch.nn.LSTM}, # Layers to quantize
dtype=torch.qint8
)
```
**Comparison**
| Aspect | Dynamic | Static |
|--------|---------|--------|
| Calibration | Not required | Required |
| Accuracy | Higher (adaptive) | Lower (fixed) |
| Speed | Moderate | Fastest |
| Latency | Variable | Consistent |
| Use Case | RNNs, variable inputs | CNNs, fixed inputs |
Dynamic quantization is the **easiest quantization method to apply** and works particularly well for recurrent models and NLP tasks where activation distributions vary significantly.