Edge AI Model Compression
# Edge AI & Model Compression
## Introduction & Motivation
Edge AI: deploy models on resource-constrained devices. Model compression; efficient inference. Applications: mobile, IoT, embedded systems.
Motivation: Reduce model size and latency for edge deployment.
Applications: Mobile inference, IoT devices, embedded systems.
---
## Core Concepts & Theory
### Quantization
Reduce precision of weights and activations.
### Pruning
Remove unimportant weights.
### Model Distillation
Transfer knowledge to smaller models.
### Low-Rank Factorization
Decompose layers into lower-rank factors.
---
## Mathematical Formulation
Quantization:
$$w_q = ext{round}( ext{scale} \cdot w) / ext{scale}$$
Magnitude Pruning:
$$ ext{mask}(w) = w ext{ if } |w| > heta ext{ else } 0$$
Compression Ratio:
$$ ext{ratio} = \frac{ ext{size}_ ext{original}}{ ext{size}_ ext{compressed}}$$
---
## Advanced Theory & Extensions
### Quantization-Aware Training (QAT)
Train with quantized weights.
### Structured Pruning
Remove entire channels or layers.
### Mixed Precision
Different bit-widths for different layers.
---
## Computational Considerations
Quantization: O(size_model).
Pruning: O(size_model·iterations).
Inference: O(size_compressed).
---
## Practical Implementation Strategies
### Post-Training Quantization
Quantize after training.
### Gradual Pruning
Progressively increase sparsity.
### Hardware-Aware Optimization
Target specific hardware.
---
## Benchmark Datasets & Evaluation
MobileNet: Efficient mobile architecture.
SqueezeNet: Lightweight CNN.
ImageNet: Standard benchmark.
---
## Key Challenges & Limitations
### Accuracy Degradation
Compression reduces performance.
### Hardware Specificity
Optimizations are hardware-dependent.
### Development Complexity
Requires specialized tools.
---
## Hyperparameter Tuning
Quantization bits: 1-8.
Pruning ratio: 0.5-0.95.
Temperature (distillation): 3-20.
---
## Real-World Applications & Case Studies
Mobile CNNs: Efficient image classification.
IoT Sensors: Minimal power consumption.
Wearables: Real-time health monitoring.
---
## Integration with Other Methods
Edge AI + knowledge distillation for accuracy preservation; + NAS for architecture optimization.
---
## Summary & Key Takeaways
Edge AI via quantization and pruning enables efficient deployment on resource-constrained devices.
Principles:
1. Quantization: Precision reduction.
2. Pruning: Sparsity.
3. Distillation: Knowledge transfer.
4. Low-rank factorization: Dimensionality reduction.
5. Hardware awareness: Device optimization.
---
---
## Appendix: Practical Labs
### Lab 1: Quantization
import numpy as np
def quantize_weights(weights, bits=8):
"""Quantize weights to fixed bit-width"""
# Get range
w_min = np.min(weights)
w_max = np.max(weights)
# Quantization levels
levels = 2 ** bits - 1
# Scale and quantize
w_scaled = (weights - w_min) / (w_max - w_min + 1e-8)
w_quant = np.round(w_scaled * levels) / levels
w_dequant = w_quant * (w_max - w_min) + w_min
return w_dequant
# Test
np.random.seed(42)
weights = np.random.randn(10, 10)
quantized = quantize_weights(weights, bits=8)
assert quantized.shape == weights.shape, "Shape preserved"
assert not np.allclose(quantized, weights), "Quantization applied"
print("✓ Quantization working")
if __name__ == "__main__":
print("Lab 1: Quantization - PASSED")### Lab 2: Magnitude Pruning
import numpy as np
def magnitude_pruning(weights, pruning_ratio=0.5):
"""Prune weights by magnitude"""
# Flatten to find threshold
flat = np.abs(weights.flatten())
threshold = np.percentile(flat, pruning_ratio * 100)
# Create mask
mask = np.abs(weights) > threshold
# Apply mask
pruned = weights * mask
return pruned, mask
# Test
np.random.seed(42)
weights = np.random.randn(10, 10)
pruned, mask = magnitude_pruning(weights, pruning_ratio=0.5)
sparsity = 1 - (np.count_nonzero(pruned) / pruned.size)
assert 0.4 < sparsity < 0.6, "Correct pruning ratio"
print("✓ Magnitude pruning working")
if __name__ == "__main__":
print("Lab 2: MagnitudePruning - PASSED")### Lab 3: Model Size Computation
import numpy as np
def compute_model_size(weights_list, bits=32):
"""Compute model size in MB"""
total_params = sum(w.size for w in weights_list)
# Size in bytes
size_bytes = total_params * (bits / 8)
# Size in MB
size_mb = size_bytes / (1024 * 1024)
return size_mb, total_params
# Test
weights = [np.random.randn(100, 100), np.random.randn(100, 10)]
size_mb_32, params = compute_model_size(weights, bits=32)
size_mb_8, _ = compute_model_size(weights, bits=8)
assert size_mb_32 > size_mb_8, "Quantization reduces size"
print("✓ Model size computation working")
if __name__ == "__main__":
print("Lab 3: ModelSizeComputation - PASSED")### Lab 4: Compression Metrics
import numpy as np
def compute_compression_metrics(original_size, compressed_size):
"""Compute compression metrics"""
compression_ratio = original_size / compressed_size
space_savings = (1 - compressed_size / original_size) * 100
return compression_ratio, space_savings
# Test
original = 100 # MB
compressed = 25 # MB
ratio, savings = compute_compression_metrics(original, compressed)
assert ratio == 4, "Compression ratio"
assert savings == 75, "Space savings"
print("✓ Compression metrics working")
if __name__ == "__main__":
print("Lab 4: CompressionMetrics - PASSED")