edge
**Edge and On-Device Inference**
**Why Edge Inference?**
Run models locally on devices for privacy, low latency, offline capability, and reduced cloud costs.
**Edge Deployment Targets**
| Target | Typical Power | Use Case |
|--------|---------------|----------|
| Mobile phones | 1-5W | Personal AI |
| Tablets | 2-10W | Field work |
| Raspberry Pi | 2-5W | IoT, prototypes |
| Jetson | 10-30W | Robotics, cameras |
| Edge servers | 100W+ | Local inference |
**Model Optimization for Edge**
**Quantization**
```python
from optimum.intel import OVQuantizer
quantizer = OVQuantizer.from_pretrained("distilbert-base-uncased")
quantizer.quantize(save_directory="./quantized_model", calibration_dataset=dataset)
```
**Pruning**
```python
import torch.nn.utils.prune as prune
# Prune 30% of weights
for module in model.modules():
if isinstance(module, torch.nn.Linear):
prune.l1_unstructured(module, name="weight", amount=0.3)
```
**Distillation**
Train smaller model to mimic larger:
```python
# Teacher: large model
# Student: small model for edge
loss = kl_div(student_logits / T, teacher_logits / T) * T^2
```
**Edge Frameworks**
| Framework | Vendor | Target |
|-----------|--------|--------|
| TensorFlow Lite | Google | Mobile, embedded |
| ONNX Runtime | Microsoft | Cross-platform |
| OpenVINO | Intel | Intel hardware |
| TensorRT | NVIDIA | NVIDIA GPUs |
| CoreML | Apple | Apple devices |
| MLC LLM | Open source | Any device |
**MLC LLM Example**
```bash
# Compile model for device
mlc_llm compile ./model --target android
# Run on Android
mlc_chat ./compiled_model
```
**Edge LLMs**
| Model | Parameters | Target |
|-------|------------|--------|
| Gemma 2B | 2B | Mobile |
| Phi-2 | 2.7B | Edge servers |
| TinyLlama | 1.1B | Embedded |
| Qwen 0.5B | 0.5B | IoT |
**Performance on Edge**
| Device | Model | Tokens/sec |
|--------|-------|------------|
| iPhone 15 Pro | Llama 7B Q4 | 10-15 |
| M2 MacBook | Llama 13B Q4 | 20-30 |
| Jetson Orin | Llama 7B Q4 | 15-25 |
**Best Practices**
- Quantize to INT4 for smallest footprint
- Use device-specific frameworks
- Profile memory and power usage
- Consider progressive loading
- Test on actual target devices