chunked prefill

Prefill and decode are the two phases of large-language-model inference, and they stress the hardware in opposite ways. Prefill processes the entire input prompt in one parallel pass to build the KV cache and emit the first token; decode then generates the rest of the answer one token at a time, each step reusing that cache. Because prefill is compute-bound and decode is memory-bandwidth-bound, modern serving systems increasingly disaggregate them onto separate GPU pools tuned to each.\n\n**Prefill is a compute-bound burst.** Reading a prompt of many tokens is a large, dense matrix multiply over all positions at once, so it keeps the GPU's arithmetic units busy and its cost scales with prompt length. This is the phase that determines time-to-first-token. A long prompt is expensive but efficient in the sense that it actually uses the compute the chip provides — the roofline sits on the compute ceiling, not the memory ceiling.\n\n**Decode is a memory-bandwidth-bound trickle.** Generating each subsequent token is a tiny matmul for a single position that must nonetheless stream the full model weights and the growing KV cache out of memory. Arithmetic intensity is low, so the GPU spends its time waiting on memory rather than computing; this phase sets the inter-token latency and dominates total time for long outputs. The two phases therefore want different things — prefill wants FLOPs, decode wants bandwidth and KV-cache capacity — and running them on the same GPU makes them fight: one big prefill can stall every in-flight decode.\n\n| | Prefill | Decode |\n|---|---|---|\n| Work per step | whole prompt, parallel | one token |\n| Bottleneck | compute (FLOPs) | memory bandwidth |\n| GPU utilization | math units saturated | mostly waiting on memory |\n| Sets | time-to-first-token | inter-token latency |\n| Scales with | prompt length | output length |\n| Wants | fast compute | bandwidth + KV capacity |\n\n```svg ```\n\n**Disaggregation runs each phase on its own pool.** Rather than time-sharing one GPU, disaggregated serving dedicates a prefill pool and a decode pool, computes the KV cache on the former, transfers it over a fast interconnect, and streams tokens from the latter. Each pool can then be sized, batched, and even built from different silicon to match its bottleneck — heavy compute for prefill, high bandwidth and memory for decode — and a burst of long prompts no longer disrupts steady token generation. It is the same divide-by-bottleneck logic behind chunked prefill, which slices long prompts so they interleave with decode instead of blocking it.\n\nRead prefill versus decode through a quant lens rather than a 'two steps' lens: they land on opposite sides of the roofline — prefill compute-bound, decode bandwidth-bound — so a single machine tuned for one is wrong for the other. Disaggregation makes the phase boundary a provisioning boundary: you scale the prefill pool by aggregate prompt FLOPs and time-to-first-token targets, and the decode pool by bandwidth, KV-cache memory, and inter-token-latency targets, and the design question becomes whether the KV-cache transfer between pools costs less than the interference you remove by separating them.