model compression for mobile

**Model compression for mobile** encompasses techniques to **reduce model size and computational requirements** so that machine learning models can run efficiently on smartphones, tablets, IoT devices, and other resource-constrained platforms. **Why Compression is Necessary** - **Memory**: Mobile devices have 4–12GB RAM shared with the OS and other apps — a 7B parameter model in FP16 requires ~14GB. - **Storage**: App store size limits and user expectations constrain model size to megabytes rather than gigabytes. - **Compute**: Mobile CPUs, GPUs, and NPUs are far less powerful than data center hardware. - **Battery**: Inference draws power — over-computation drains batteries and generates heat. - **Latency**: Users expect instant responses — model must be fast enough for real-time interaction. **Compression Techniques** - **Quantization**: Reduce numerical precision from FP32 → FP16 → INT8 → INT4. Cuts model size by 2–8× with minimal quality loss. INT4 quantization is commonly used for on-device LLMs. - **Pruning**: Remove redundant weights (near-zero values) or entire neurons/attention heads. **Structured pruning** removes entire channels for hardware-friendly speedups. - **Knowledge Distillation**: Train a small "student" model to mimic a large "teacher" model. The student is compact but retains much of the teacher's capability. - **Architecture Optimization**: Use efficient architectures designed for mobile — **MobileNet**, **EfficientNet**, **SqueezeNet** for vision; **TinyLlama**, **Phi-3-mini** for language. - **Weight Sharing**: Multiple network connections share the same weight value, reducing unique parameters. - **Low-Rank Factorization**: Decompose large weight matrices into products of smaller matrices, reducing parameters. **Mobile-Specific Optimizations** - **Operator Fusion**: Combine multiple operations (convolution + batch norm + activation) into a single optimized kernel. - **Hardware-Aware Optimization**: Optimize for specific hardware features (Apple Neural Engine, Qualcomm Hexagon DSP, Google TPU in Pixel). - **Dynamic Shapes**: Handle variable input sizes efficiently without padding waste. **Frameworks**: **TensorFlow Lite**, **Core ML**, **ONNX Runtime**, **NCNN**, **MNN**, **ExecuTorch**. **Current State**: On-device LLMs (3B–7B parameters with 4-bit quantization) now run on flagship smartphones, enabling local assistants, text generation, and code completion without cloud connectivity. Model compression is the **enabling technology** for on-device AI — without it, modern neural networks are simply too large for mobile deployment.