static quantization

**Static quantization** uses **fixed quantization parameters** (scale and zero-point) determined during a calibration phase, rather than computing them dynamically at runtime. Both weights and activations are quantized using these pre-determined parameters. **How It Works** 1. **Calibration**: Run the model on a representative calibration dataset (typically 100-1000 samples) to observe the range of activation values in each layer. 2. **Parameter Determination**: Compute scale and zero-point for each activation tensor based on observed min/max values (or percentiles to handle outliers). 3. **Quantization**: Quantize both weights and activations using the fixed parameters. 4. **Inference**: All operations (matrix multiplications, convolutions) are performed in INT8 using the pre-determined quantization parameters. **Advantages** - **Maximum Speed**: No runtime overhead for computing quantization parameters — all operations are pure INT8 arithmetic. - **Consistent Latency**: Inference time is deterministic and predictable. - **Hardware Optimization**: Fully compatible with INT8-optimized hardware accelerators (TPUs, NPUs, DSPs). - **Maximum Compression**: Both weights and activations are quantized, minimizing memory bandwidth. **Disadvantages** - **Calibration Required**: Needs a representative calibration dataset that covers the expected input distribution. - **Fixed Parameters**: Cannot adapt to inputs outside the calibration range — may lose accuracy on out-of-distribution inputs. - **Accuracy Loss**: Typically 1-5% accuracy drop compared to FP32, though quantization-aware training can recover most of this. **Calibration Strategies** - **Min-Max**: Use the absolute min/max observed during calibration. Simple but sensitive to outliers. - **Percentile**: Use 0.1% and 99.9% percentiles to clip outliers. More robust. - **Entropy (KL Divergence)**: Minimize the information loss between FP32 and INT8 distributions. Used by TensorRT. - **MSE**: Minimize mean squared error between FP32 and INT8 activations. **When to Use Static Quantization** - **Production Deployment**: When maximum inference speed is critical. - **Edge Devices**: When deploying to resource-constrained hardware. - **CNNs**: Convolutional networks with relatively stable activation distributions. - **Known Input Distribution**: When the deployment input distribution matches the calibration data. Static quantization is the **standard choice for production deployment** of CNNs and other models where maximum inference speed and hardware compatibility are priorities.