Efficient Neural Network Inference is the systems engineering discipline that minimizes the computational cost, memory footprint, and latency of deploying trained neural networks — through complementary techniques including quantization (FP32→INT8/INT4), pruning (removing redundant parameters), knowledge distillation (training small student from large teacher), and architecture optimization (MobileNet, EfficientNet), enabling deployment on resource-constrained devices from smartphones to microcontrollers while maintaining task-relevant accuracy.

Quantization

Replace high-precision floating-point weights and activations with lower-precision fixed-point representations:

• FP32 → FP16/BF16: 2× memory reduction, 2× compute speedup on hardware with FP16 units. Negligible accuracy loss for most models.
• FP32 → INT8: 4× memory reduction, 2-4× speedup on INT8 hardware (all modern CPUs and GPUs). Post-training quantization (PTQ): calibrate scale/zero-point on a representative dataset. Quantization-aware training (QAT): simulate quantization during training for higher accuracy.
• INT4/INT3: 8-10× compression of large language models (GPTQ, AWQ, GGML). Requires careful weight selection — salient weights (high-magnitude, significant for accuracy) kept at higher precision.

Pruning

Remove parameters that contribute least to model accuracy:

• Unstructured Pruning: Zero out individual weights below a threshold. Achieves 90%+ sparsity on many models with minimal accuracy loss. Requires sparse computation hardware/software for actual speedup (dense hardware ignores zeros but still computes them).
• Structured Pruning: Remove entire channels, attention heads, or layers. Produces a smaller dense model that runs faster on standard hardware without sparse support. Typically achieves 2-4× speedup with 1-2% accuracy loss.

Knowledge Distillation

Train a small "student" model to mimic a large "teacher" model:

• Logit Distillation: Student trained on soft targets (teacher's output probabilities at high temperature). Dark knowledge in inter-class relationships transfers — the teacher's distribution over wrong classes encodes similarity structure.
• Feature Distillation: Student trained to match teacher's intermediate feature maps. Richer signal than logits alone.
• DistilBERT: 6 layers distilled from BERT's 12 layers. 40% smaller, 60% faster, retains 97% of BERT's accuracy on GLUE benchmarks.

Efficient Architectures

• MobileNet (v1-v3): Depthwise separable convolutions reduce FLOPs by 8-9× vs. standard convolution at similar accuracy. Designed for mobile deployment.
• EfficientNet: Compound scaling of depth, width, and resolution simultaneously. EfficientNet-B0: 5.3M params, 77.1% ImageNet top-1. EfficientNet-B7: 66M params, 84.3%.
• TinyML: Models for microcontrollers with <1 MB RAM: MCUNet, TinyNN. Run image classification on ARM Cortex-M at <1 ms latency.

Inference Frameworks

• TensorRT (NVIDIA): Optimizes and deploys models on NVIDIA GPUs. Layer fusion, precision calibration, kernel auto-tuning. 2-5× speedup over PyTorch inference.
• ONNX Runtime: Cross-platform inference. Optimizations for CPU (Intel, ARM), GPU, and NPU.
• TFLite / Core ML: Mobile inference on Android/iOS with hardware acceleration (GPU, Neural Engine, NPU).

Efficient Inference is the deployment engineering that converts research models into production reality — the techniques that bridge the gap between training-time model quality and the compute, memory, and latency constraints of real-world deployment environments.