model serving inference

**ML Model Serving and Inference Optimization** is the **engineering discipline of deploying trained models into production systems that process real-time requests at scale — where the challenges shift from training accuracy to inference latency, throughput, cost, and reliability, requiring specialized optimization techniques (quantization, batching, graph optimization, hardware-specific compilation) to achieve millisecond-level response times at thousands of requests per second**. **The Inference Challenge** Training is batch-oriented (maximize GPU utilization over hours/days). Inference is request-oriented (minimize latency for each query while maximizing throughput). A model that takes 50 ms per request on a V100 GPU needs to serve 1,000 requests/second — requiring batching, pipelining, multi-GPU deployment, and aggressive optimization. **Model Optimization Techniques** - **Quantization**: Reduce weight and activation precision from FP32 to FP16/INT8/INT4. Post-Training Quantization (PTQ) converts a trained model without retraining. INT8 quantization provides ~2-4x speedup on GPUs (Tensor Core INT8) and CPUs (VNNI). For LLMs, GPTQ and AWQ achieve 4-bit quantization with minimal quality loss. - **Graph Optimization**: Fuse operations (Conv+BN+ReLU → single kernel), eliminate redundant operations, constant folding. TensorRT, ONNX Runtime, and XLA apply these automatically. - **Pruning**: Remove weights (unstructured) or entire neurons/channels (structured) that contribute minimally to output. Structured pruning directly reduces computation; unstructured pruning requires sparse-aware hardware. - **Knowledge Distillation**: Train a smaller model to mimic the larger one. DistilBERT is 60% the size, 2x faster, 97% accuracy of BERT. **Serving Frameworks** - **NVIDIA Triton Inference Server**: Multi-framework (PyTorch, TensorFlow, ONNX, TensorRT), multi-model serving with dynamic batching, model ensembles, and GPU sharing. The standard for GPU-based inference at scale. - **ONNX Runtime**: Cross-platform inference engine. Export models to ONNX format, optimize with graph transformations and execution providers (CUDA, TensorRT, DirectML, CoreML). Single model format deployable across GPUs, CPUs, and edge devices. - **vLLM**: High-throughput LLM serving engine using PagedAttention for memory-efficient KV cache management. Achieves 2-4x higher throughput than naive HuggingFace serving. - **TensorRT-LLM**: NVIDIA's optimized LLM inference library. In-flight batching, quantization-aware kernels, tensor parallelism for multi-GPU LLM serving. **Key Serving Patterns** - **Dynamic Batching**: Accumulate incoming requests and batch them together for GPU processing. Wait up to a configurable deadline (e.g., 10 ms) to form larger batches. Throughput increases dramatically (8x-32x) with batching at modest latency cost. - **Continuous/In-Flight Batching**: For autoregressive LLMs, new requests join the batch as existing requests complete tokens. Avoids waiting for the longest sequence in the batch to finish. vLLM and TensorRT-LLM implement this. - **Model Parallelism for Serving**: Large models that exceed single-GPU memory are split across GPUs using tensor or pipeline parallelism. Inference-time parallelism trades latency for the ability to serve models that won't fit on one device. ML Model Serving is **the bridge between trained models and real-world impact** — the engineering that transforms a research artifact consuming GPU-hours for a single prediction into a production system handling millions of requests per day at sub-100-millisecond latency and dollars-per-million-requests cost.