Model Compression Pruning Efficient Neural Networks
# Model Compression & Pruning: Efficient Neural Networks
## Introduction & Motivation
Model compression: reduce model size/inference cost. Pruning: remove weights; structured or unstructured. Knowledge distillation: student-teacher knowledge transfer. Quantization: reduce precision; 8-bit or 4-bit. Factorization: decompose layers. Applications: mobile/edge deployment, latency reduction, resource constraints.
Motivation: Large models slow on edge. Compression enables deployment.
Applications: Mobile, edge devices, real-time inference.
---
## Core Concepts & Theory
### Weight Pruning
Remove small-magnitude weights; sparse network.
### Structured Pruning
Remove entire channels/filters; hardware efficient.
### Knowledge Distillation
Train small student on large teacher; knowledge transfer.
---
## Mathematical Formulation
Pruning threshold:
$$w_i = \begin{cases} 0 & |w_i| < heta \\ w_i & |w_i| \geq heta \end{cases}$$
Distillation loss:
$$L = \alpha L_{ ext{CE}}(y, \hat{y}_s) + (1-\alpha) L_{ ext{KL}}(p_t, p_s)$$
where p = softmax(logits/T), T = temperature.
---
## Advanced Theory & Extensions
### Lottery Ticket Hypothesis
Pruned networks as good as original; find "winning tickets".
### Neural Architecture Search
Automatically find efficient architectures.
### Low-Rank Factorization
Decompose weight matrices; reduce parameters.
---
## Computational Considerations
Pruning: O(1) per weight; sparse operations slow.
Distillation: O(teacher + student) inference.
Quantization: O(1) operations; reduced precision.
---
## Practical Implementation Strategies
### Gradual Pruning
Prune iteratively; fine-tune between steps.
### Layer-Wise Sensitivity
Different layers; different pruning rates.
### Post-Training Quantization
Quantize after training; simpler pipeline.
---
## Benchmark Datasets & Evaluation
ImageNet: Pruning 90%+ parameters; minimal accuracy loss.
Mobile: 10-100MB model size target.
Real-Time: <100ms latency requirement.
---
## Key Challenges & Limitations
### Accuracy-Efficiency Tradeoff
More compression → lower accuracy.
### Hardware Dependency
Sparse operations; not all hardware efficient.
### Retraining Cost
Pruning requires fine-tuning; compute intensive.
---
## Hyperparameter Tuning
Pruning rate: 80-95% typical; empirical.
Distillation temperature: 3-20; higher = softer.
Student capacity: 10-50% of teacher.
---
## Real-World Applications & Case Studies
Mobile: TensorFlow Lite; quantized mobilenet.
Edge: Pruned ResNets; on-device inference.
IoT: Extreme compression; 1-10MB models.
---
## Integration with Other Methods
Compression + Quantization → extreme efficiency.
Compression + NAS → efficient architecture search.
---
## Summary & Key Takeaways
Model compression via pruning, distillation, and quantization reduces model size and latency while maintaining performance, enabling efficient deployment.
Principles:
1. Weight pruning: remove small-magnitude weights.
2. Structured: channel/filter removal; hardware efficient.
3. Distillation: knowledge transfer via soft targets.
4. Quantization: reduce precision.
5. Lottery ticket: sparse networks sufficient.
---
---
## Appendix: Practical Labs
### Lab 1: Weight Pruning
import torch
import numpy as np
def prune_weights(model, pruning_rate=0.9):
"""Simple weight pruning by magnitude"""
for name, param in model.named_parameters():
if 'weight' in name and len(param.shape) > 1:
threshold = np.percentile(param.data.abs().numpy(), pruning_rate * 100)
mask = param.data.abs() > threshold
param.data = param.data * mask
return model
# Test
np.random.seed(42)
model = torch.nn.Sequential(
torch.nn.Linear(10, 20),
torch.nn.ReLU(),
torch.nn.Linear(20, 5)
)
pruned = prune_weights(model, pruning_rate=0.9)
total_params = sum(p.numel() for p in model.parameters())
pruned_params = sum((p.data != 0).sum() for p in model.parameters() if p.dim() > 1)
print("✓ Weight pruning working")
if __name__ == "__main__":
print("Lab 1: Pruning - PASSED")### Lab 2: Knowledge Distillation Loss
import torch
import torch.nn.functional as F
import numpy as np
def distillation_loss(student_logits, teacher_logits, labels, temperature=4.0, alpha=0.5):
"""Compute distillation loss"""
# Cross-entropy on student
ce_loss = F.cross_entropy(student_logits, labels)
# KL divergence between teacher and student
teacher_probs = F.softmax(teacher_logits / temperature, dim=1)
student_log_probs = F.log_softmax(student_logits / temperature, dim=1)
kl_loss = F.kl_div(student_log_probs, teacher_probs, reduction='mean')
# Combined loss
loss = alpha * ce_loss + (1 - alpha) * kl_loss * (temperature ** 2)
return loss
# Test
np.random.seed(42)
student_logits = torch.randn(32, 10)
teacher_logits = torch.randn(32, 10)
labels = torch.randint(0, 10, (32,))
loss = distillation_loss(student_logits, teacher_logits, labels)
assert torch.isfinite(loss), "Loss finite"
assert loss >= 0, "Loss non-negative"
print("✓ Distillation loss working")
if __name__ == "__main__":
print("Lab 2: Distillation - PASSED")### Lab 3: Quantization
import numpy as np
def quantize_int8(weights, scale=None):
"""Simple int8 quantization"""
if scale is None:
scale = np.abs(weights).max() / 127
quantized = np.round(weights / scale).astype(np.int8)
return quantized, scale
def dequantize_int8(quantized, scale):
"""Dequantization"""
return quantized.astype(np.float32) * scale
# Test
np.random.seed(42)
weights = np.random.randn(100, 100)
quantized, scale = quantize_int8(weights)
reconstructed = dequantize_int8(quantized, scale)
error = np.mean(np.abs(weights - reconstructed))
assert error < 0.1, "Reconstruction error reasonable"
print("✓ Quantization working")
if __name__ == "__main__":
print("Lab 3: Quantization - PASSED")### Lab 4: Sparsity Measurement
import numpy as np
def measure_sparsity(weights):
"""Compute weight sparsity"""
zero_count = np.count_nonzero(weights == 0)
total_count = weights.size
sparsity = zero_count / total_count
return sparsity
# Test
np.random.seed(42)
weights = np.random.randn(100, 100)
# Prune some weights
weights[weights.abs() < 0.5] = 0
sparsity = measure_sparsity(weights)
assert 0 <= sparsity <= 1, "Sparsity in [0,1]"
print("✓ Sparsity measurement working")
if __name__ == "__main__":
print("Lab 4: Sparsity - PASSED")