Dynamic quantization determines quantization parameters (scale and zero-point) at runtime based on the actual values flowing through the network during inference, rather than using fixed parameters determined during calibration.

How It Works

• Weights: Quantized statically (ahead of time) and stored in INT8 format.
• Activations: Remain in floating-point during computation. Quantization parameters are computed dynamically for each batch based on the observed min/max values.
• Computation: Matrix multiplications and other operations are performed in INT8, but activations are quantized on-the-fly.

Workflow

1. Load: Load pre-quantized INT8 weights. 2. Observe: For each activation tensor, compute min/max values from the current batch. 3. Quantize: Compute scale and zero-point, quantize activations to INT8. 4. Compute: Perform INT8 operations (e.g., matrix multiplication). 5. Dequantize: Convert results back to FP32 for the next layer.

Advantages

• No Calibration: No need for a calibration dataset to determine activation ranges — the model adapts to the actual input distribution at runtime.
• Accuracy: Often achieves better accuracy than static quantization because it adapts to each input's specific value range.
• Easy to Apply: Can be applied post-training without retraining or fine-tuning.

Disadvantages

• Runtime Overhead: Computing min/max and quantization parameters for each batch adds latency (typically 10-30% slower than static quantization).
• Variable Latency: Inference time varies depending on input value ranges.
• Limited Speedup: Activations are quantized/dequantized repeatedly, reducing the efficiency gains compared to static quantization.

When to Use Dynamic Quantization

• Recurrent Models: LSTMs, GRUs, and Transformers where activation ranges vary significantly across sequences.
• Variable Input Distributions: When inputs have unpredictable value ranges (e.g., user-generated content).
• Quick Deployment: When you need quantization benefits without the effort of calibration.

PyTorch Example

import torch

model = MyModel()
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear, torch.nn.LSTM},  # Layers to quantize
    dtype=torch.qint8
)

Comparison

Dynamic quantization is the easiest quantization method to apply and works particularly well for recurrent models and NLP tasks where activation distributions vary significantly.