logarithmic quantization
**Logarithmic quantization** applies quantization on a **logarithmic scale** rather than a linear scale, allocating more precision to smaller values and less precision to larger values. This approach is particularly effective for neural network weights and activations that follow exponential or power-law distributions.
**How It Works**
- **Linear Quantization**: Divides the value range into equal intervals. A value of 0.1 and 0.2 get the same precision as 10.0 and 10.1.
- **Logarithmic Quantization**: Divides the **logarithmic space** into equal intervals. Smaller values (near zero) receive finer granularity, while larger values are coarsely quantized.
**Mathematical Representation**
For a value $x$, logarithmic quantization computes:
$$q = ext{round}(log_2(|x|) cdot s) cdot ext{sign}(x)$$
Where $s$ is a scale factor. Dequantization reconstructs:
$$hat{x} = 2^{q/s} cdot ext{sign}(x)$$
**Advantages**
- **Better Dynamic Range**: Captures both very small and very large values effectively without wasting quantization levels.
- **Natural Fit for Weights**: Neural network weights often follow distributions where most values are small, making logarithmic quantization more efficient than linear.
- **Reduced Quantization Error**: For exponentially distributed data, logarithmic quantization minimizes mean squared error compared to linear quantization.
**Applications**
- **Model Compression**: Quantize weights in deep networks where weight magnitudes span several orders of magnitude.
- **Audio Processing**: Audio signals have logarithmic perceptual characteristics (decibels), making log quantization natural.
- **Gradient Compression**: Gradients in distributed training often have exponential distributions.
**Comparison to Linear Quantization**
| Aspect | Linear | Logarithmic |
|--------|--------|-------------|
| Precision Distribution | Uniform across range | Higher for small values |
| Dynamic Range | Limited | Excellent |
| Implementation | Simple | Slightly more complex |
| Best For | Uniform distributions | Exponential distributions |
Logarithmic quantization is less common than linear quantization but provides significant advantages for specific data distributions, particularly in model compression and audio applications.