Model compression for mobile encompasses techniques to reduce model size and computational requirements so that machine learning models can run efficiently on smartphones, tablets, IoT devices, and other resource-constrained platforms.

Why Compression is Necessary

• Memory: Mobile devices have 4–12GB RAM shared with the OS and other apps — a 7B parameter model in FP16 requires ~14GB.
• Storage: App store size limits and user expectations constrain model size to megabytes rather than gigabytes.
• Compute: Mobile CPUs, GPUs, and NPUs are far less powerful than data center hardware.
• Battery: Inference draws power — over-computation drains batteries and generates heat.
• Latency: Users expect instant responses — model must be fast enough for real-time interaction.

Compression Techniques

• Quantization: Reduce numerical precision from FP32 → FP16 → INT8 → INT4. Cuts model size by 2–8× with minimal quality loss. INT4 quantization is commonly used for on-device LLMs.
• Pruning: Remove redundant weights (near-zero values) or entire neurons/attention heads. Structured pruning removes entire channels for hardware-friendly speedups.
• Knowledge Distillation: Train a small "student" model to mimic a large "teacher" model. The student is compact but retains much of the teacher's capability.
• Architecture Optimization: Use efficient architectures designed for mobile — MobileNet, EfficientNet, SqueezeNet for vision; TinyLlama, Phi-3-mini for language.
• Weight Sharing: Multiple network connections share the same weight value, reducing unique parameters.
• Low-Rank Factorization: Decompose large weight matrices into products of smaller matrices, reducing parameters.

Mobile-Specific Optimizations

• Operator Fusion: Combine multiple operations (convolution + batch norm + activation) into a single optimized kernel.
• Hardware-Aware Optimization: Optimize for specific hardware features (Apple Neural Engine, Qualcomm Hexagon DSP, Google TPU in Pixel).
• Dynamic Shapes: Handle variable input sizes efficiently without padding waste.

Frameworks: TensorFlow Lite, Core ML, ONNX Runtime, NCNN, MNN, ExecuTorch.

Current State: On-device LLMs (3B–7B parameters with 4-bit quantization) now run on flagship smartphones, enabling local assistants, text generation, and code completion without cloud connectivity.

Model compression is the enabling technology for on-device AI — without it, modern neural networks are simply too large for mobile deployment.