continuous batching

Continuous batching (also called iteration-level or in-flight batching) is a serving strategy for large language models that rebuilds the batch every decoding step instead of once per batch. As soon as any request in the batch finishes generating, its place is given to a request waiting in the queue, so the GPU keeps processing a full batch of useful work rather than idling while it waits for the slowest request to complete.\n\n**Static batching stalls on mixed request lengths.** A conventional batch launches a fixed group of requests together and holds them until every one is done. Because LLM outputs vary wildly in length, a request that emits its stop token after three tokens still occupies its slot while a neighbor generates hundreds more. Those freed slots sit idle, and no new request can be admitted until the entire batch retires, so effective throughput drops sharply exactly when traffic is a realistic mix of short and long generations.\n\n**Continuous batching schedules at the token, not the batch.** The scheduler re-evaluates the active set on every forward pass: finished sequences are evicted immediately and pending sequences are injected into the newly free slots. The GPU therefore runs a nearly full batch each step regardless of how the individual request lengths line up. This is the serving-layer complement to the memory tricks it pairs with — it keeps the compute busy, while PagedAttention keeps the KV-cache memory that those extra concurrent requests need from fragmenting.\n\n| | Static batching | Continuous batching |\n|---|---|---|\n| Scheduling unit | whole batch | each decode step |\n| Finished request | holds its slot | evicted immediately |\n| New request admitted | only at batch start | into any freed slot |\n| GPU on mixed lengths | idles on short reqs | stays near-full |\n| Throughput | low under variance | high, length-robust |\n| Tail latency | coupled to slowest | decoupled per request |\n\n```svg\n\n```\n\n**It is why modern inference servers hit high throughput.** Frameworks like vLLM, TensorRT-LLM, and TGI make continuous batching the default because it converts idle GPU time directly into served tokens, often several-fold, without touching model quality. It also improves fairness and latency: a short request no longer has to wait behind a long one in the same batch, since it can retire the instant it finishes. The gains are largest under bursty, heterogeneous traffic — the normal condition for a production API.\n\nRead continuous batching through a quant lens rather than a 'better batching' lens: the metric it moves is GPU utilization under length variance, and the mechanism is turning batch scheduling from a per-batch decision into a per-iteration one. The design question is how much your request lengths vary and how bursty arrivals are — the wider that distribution, the more idle slots static batching leaves and the more continuous batching recovers, up to the point where KV-cache memory, not compute scheduling, becomes the binding constraint on how many requests you can keep in flight.