Static quantization uses fixed quantization parameters (scale and zero-point) determined during a calibration phase, rather than computing them dynamically at runtime. Both weights and activations are quantized using these pre-determined parameters.

How It Works

1. Calibration: Run the model on a representative calibration dataset (typically 100-1000 samples) to observe the range of activation values in each layer. 2. Parameter Determination: Compute scale and zero-point for each activation tensor based on observed min/max values (or percentiles to handle outliers). 3. Quantization: Quantize both weights and activations using the fixed parameters. 4. Inference: All operations (matrix multiplications, convolutions) are performed in INT8 using the pre-determined quantization parameters.

Advantages

• Maximum Speed: No runtime overhead for computing quantization parameters — all operations are pure INT8 arithmetic.
• Consistent Latency: Inference time is deterministic and predictable.
• Hardware Optimization: Fully compatible with INT8-optimized hardware accelerators (TPUs, NPUs, DSPs).
• Maximum Compression: Both weights and activations are quantized, minimizing memory bandwidth.

Disadvantages

• Calibration Required: Needs a representative calibration dataset that covers the expected input distribution.
• Fixed Parameters: Cannot adapt to inputs outside the calibration range — may lose accuracy on out-of-distribution inputs.
• Accuracy Loss: Typically 1-5% accuracy drop compared to FP32, though quantization-aware training can recover most of this.

Calibration Strategies

• Min-Max: Use the absolute min/max observed during calibration. Simple but sensitive to outliers.
• Percentile: Use 0.1% and 99.9% percentiles to clip outliers. More robust.
• Entropy (KL Divergence): Minimize the information loss between FP32 and INT8 distributions. Used by TensorRT.
• MSE: Minimize mean squared error between FP32 and INT8 activations.

When to Use Static Quantization

• Production Deployment: When maximum inference speed is critical.
• Edge Devices: When deploying to resource-constrained hardware.
• CNNs: Convolutional networks with relatively stable activation distributions.
• Known Input Distribution: When the deployment input distribution matches the calibration data.

Static quantization is the standard choice for production deployment of CNNs and other models where maximum inference speed and hardware compatibility are priorities.