Edge and On-Device Inference

Why Edge Inference? Run models locally on devices for privacy, low latency, offline capability, and reduced cloud costs.

Edge Deployment Targets

Model Optimization for Edge

Quantization

from optimum.intel import OVQuantizer

quantizer = OVQuantizer.from_pretrained("distilbert-base-uncased")
quantizer.quantize(save_directory="./quantized_model", calibration_dataset=dataset)

Pruning

import torch.nn.utils.prune as prune

# Prune 30% of weights
for module in model.modules():
    if isinstance(module, torch.nn.Linear):
        prune.l1_unstructured(module, name="weight", amount=0.3)

Distillation Train smaller model to mimic larger:

# Teacher: large model
# Student: small model for edge
loss = kl_div(student_logits / T, teacher_logits / T) * T^2

Edge Frameworks

MLC LLM Example

# Compile model for device
mlc_llm compile ./model --target android

# Run on Android
mlc_chat ./compiled_model

Edge LLMs

Performance on Edge

Best Practices

• Quantize to INT4 for smallest footprint
• Use device-specific frameworks
• Profile memory and power usage
• Consider progressive loading
• Test on actual target devices