distributed inference serving

**Distributed Inference Serving** is the **systems engineering discipline of deploying large neural network models across multiple GPUs, multiple machines, or heterogeneous accelerator fleets to serve real-time prediction requests at production-grade latency, throughput, and availability — solving the fundamental problem that frontier models are too large for any single device**. **Why Single-GPU Inference Breaks** A 70B-parameter model in FP16 requires 140 GB of VRAM just for weights — more than any single GPU offers. Even models that fit in memory face throughput walls: a single GPU serving a chatbot to 1,000 concurrent users would queue requests for minutes. Distributed inference splits the model and the workload across devices. **Distribution Strategies** - **Tensor Parallelism (TP)**: Each layer's weight matrix is split across GPUs. For a linear layer Y = XW, W is partitioned column-wise or row-wise, each GPU computes its shard, and an all-reduce synchronizes the partial results. Requires fast interconnect (NVLink/NVSwitch) because synchronization happens at every layer. - **Pipeline Parallelism (PP)**: Different layers are assigned to different GPUs. GPU 0 runs layers 1-20, GPU 1 runs layers 21-40, etc. Request microbatches pipeline through the stages. Higher latency for individual requests but good throughput with many concurrent requests. - **Data Parallelism / Replication**: Multiple identical copies of the model serve different requests simultaneously. A load balancer routes incoming requests to the least-loaded replica. Scales throughput linearly with replicas but multiplies memory cost. **Continuous Batching and PagedAttention** Modern inference servers (vLLM, TensorRT-LLM, TGI) use continuous batching: instead of waiting for all requests in a batch to finish, new requests are inserted as soon as any slot opens. PagedAttention (vLLM) manages the KV cache as virtual memory pages, eliminating the massive memory waste from pre-allocated, fixed-length KV cache slots. **Optimization Stack** - **Speculative Decoding**: A small draft model generates candidate tokens quickly; the large target model verifies them in parallel. When the draft is accurate, multiple tokens are accepted per forward pass, reducing effective latency. - **Quantization**: INT8/INT4 quantization halves or quarters the memory footprint, allowing larger batch sizes and reducing inter-GPU communication volume. - **Prefix Caching**: For applications where many requests share a common system prompt, the KV cache for the shared prefix is computed once and reused across all requests. Distributed Inference Serving is **the infrastructure layer that makes frontier AI models accessible as real-time services** — transforming massive research checkpoints from offline batch-processing artifacts into responsive, concurrent production endpoints.