Logarithmic quantization applies quantization on a logarithmic scale rather than a linear scale, allocating more precision to smaller values and less precision to larger values. This approach is particularly effective for neural network weights and activations that follow exponential or power-law distributions.

How It Works

• Linear Quantization: Divides the value range into equal intervals. A value of 0.1 and 0.2 get the same precision as 10.0 and 10.1.
• Logarithmic Quantization: Divides the logarithmic space into equal intervals. Smaller values (near zero) receive finer granularity, while larger values are coarsely quantized.

Mathematical Representation

For a value $x$, logarithmic quantization computes:

Where $s$ is a scale factor. Dequantization reconstructs:

Advantages

• Better Dynamic Range: Captures both very small and very large values effectively without wasting quantization levels.
• Natural Fit for Weights: Neural network weights often follow distributions where most values are small, making logarithmic quantization more efficient than linear.
• Reduced Quantization Error: For exponentially distributed data, logarithmic quantization minimizes mean squared error compared to linear quantization.

Applications

• Model Compression: Quantize weights in deep networks where weight magnitudes span several orders of magnitude.
• Audio Processing: Audio signals have logarithmic perceptual characteristics (decibels), making log quantization natural.
• Gradient Compression: Gradients in distributed training often have exponential distributions.

Comparison to Linear Quantization

Logarithmic quantization is less common than linear quantization but provides significant advantages for specific data distributions, particularly in model compression and audio applications.