inference

LLM inference is the serving side of a large language model: taking a trained model and running its forward pass to answer real user requests, optimized for latency, throughput, and cost. Training happens once; inference happens on every request, forever, which is why it dominates the operational cost of any deployed model and why so much engineering goes into making it faster and cheaper.\n\n```svg\n\n```\n\n**Inference has two phases with opposite characters.** A request is served in a *prefill* phase that reads the entire prompt in one parallel pass, and a *decode* phase that then generates the answer one token at a time. Prefill is compute-bound — it is a batch of large matrix multiplies over all prompt tokens — and it sets the time to first token (TTFT). Decode is memory-bound — every new token requires a full forward pass that reads the growing KV cache — and its per-token cost sets the time per output token (TPOT). Almost every inference optimization is aimed at one of these two phases.\n\n**Autoregression is the reason decode is slow.** The model cannot produce token five until it has produced token four, so decode is inherently sequential. Each step does relatively little arithmetic but must stream the full model weights and the entire KV cache through the GPU, so the bottleneck is memory bandwidth, not compute. This is why a single request leaves most of a GPU's arithmetic units idle, and why serving is fundamentally a batching problem.\n\n**Batching is what makes serving economical.** Because decode underuses the GPU, the way to get throughput is to run many requests together so their token-generation steps share each weight read. *Continuous batching* (also called in-flight batching) adds and removes requests from the running batch every step instead of waiting for a whole batch to finish, keeping the GPU full even as sequences start and end at different times. This single technique is the largest throughput lever in modern serving.\n\n**The KV cache is the central resource.** Every token's keys and values are cached so they are not recomputed, but that cache grows with sequence length and batch size and quickly becomes the memory bottleneck. *PagedAttention* manages it in fixed pages like virtual memory to eliminate fragmentation, quantized KV cache shrinks it to INT8 or INT4, and grouped-query attention (GQA/MQA) reduces how many KV heads must be stored at all. How much cache you can hold directly sets how many requests you can batch.\n\n**The rest of the toolkit attacks one phase or the other.** Quantizing weights to INT8, FP8, or INT4 cuts the memory traffic that throttles decode; FlashAttention fuses the attention kernel to avoid round-trips to slow memory; tensor parallelism splits a model too big for one GPU; speculative decoding lets a small draft model propose several tokens that the big model verifies in one pass, breaking the one-token-per-step limit. Frameworks like vLLM, TensorRT-LLM, TGI, and SGLang package these together behind an API.\n\n| Phase | Bottleneck | Latency scales with | Key metric | Main levers |\n|---|---|---|---|---|\n| Prefill | GPU compute (matmul) | prompt length | TTFT | FlashAttention, tensor parallelism, prefix caching |\n| Decode | memory bandwidth (KV + weight reads) | output length | TPOT | continuous batching, quantization, speculative decoding |\n\n| Metric | Meaning | Who cares |\n|---|---|---|\n| TTFT | time to first token | interactive feel, chat responsiveness |\n| TPOT | time per output token | streaming speed, long-answer latency |\n| Throughput | tokens/sec across all requests | cost per token, GPU utilization |\n\nRead inference through a *prefill-versus-decode* lens rather than a *single-number-latency* lens: the two phases are bound by different resources, so they respond to different optimizations, and the whole discipline of serving is deciding how to trade one against the other. Prefill wants compute and parallelism; decode wants memory bandwidth and big batches — and every framework, from vLLM to TensorRT-LLM, is ultimately a set of choices about how to keep both phases fed while packing enough concurrent requests onto the GPU to make the economics work.\n