continuous batching
**Continuous Batching** is an **LLM serving optimization that dynamically inserts new requests into a running inference batch as soon as individual sequences complete** — replacing static batching (where the entire batch waits for the longest sequence to finish) with iteration-level scheduling that fills freed GPU capacity immediately, achieving up to 20× higher throughput by eliminating the GPU idle time caused by variable-length sequence generation.
**What Is Continuous Batching?**
- **Definition**: A scheduling strategy for LLM inference where the serving system operates at the granularity of individual decoding iterations rather than complete requests — when one sequence in the batch finishes generating (hits the end-of-sequence token), a new request from the queue immediately takes its slot in the next iteration, keeping the GPU fully utilized.
- **Static Batching Problem**: In static (naive) batching, a batch of N requests starts together and finishes only when the longest sequence completes — if one request generates 10 tokens and another generates 2000 tokens, the GPU sits idle for the short request's slot during 1990 iterations.
- **Iteration-Level Scheduling**: Continuous batching makes scheduling decisions at every decoding step — checking if any sequence has finished, removing completed sequences, and inserting waiting requests into the freed slots.
- **Also Called**: In-flight batching, dynamic batching, or iteration-level batching — all refer to the same concept of per-iteration request management.
**Why Continuous Batching Matters**
- **Throughput**: Continuous batching achieves 5-20× higher throughput than static batching for workloads with variable output lengths — the improvement is proportional to the variance in sequence lengths.
- **Latency Fairness**: Short requests complete quickly without waiting for long requests in the same batch — eliminating "head-of-line blocking" where a single long generation delays all other requests.
- **GPU Utilization**: Keeps GPU compute units occupied at every iteration — static batching wastes GPU cycles on padding tokens for completed sequences, while continuous batching fills those slots with real work.
- **Cost Efficiency**: Higher throughput per GPU means fewer GPUs needed to serve the same request volume — directly reducing infrastructure cost for LLM serving.
**Continuous Batching with PagedAttention**
- **Memory Challenge**: Each active request maintains a KV cache that grows with sequence length — continuous batching requires efficient memory management to handle requests entering and leaving the batch dynamically.
- **PagedAttention (vLLM)**: Manages KV cache memory like virtual memory pages — allocating and freeing cache blocks dynamically as requests enter and leave the batch, eliminating memory fragmentation.
- **Memory Efficiency**: PagedAttention + continuous batching achieves near-zero memory waste — compared to static batching which must pre-allocate maximum sequence length for every request.
| Feature | Static Batching | Continuous Batching |
|---------|----------------|-------------------|
| Scheduling Granularity | Per-batch | Per-iteration |
| GPU Utilization | Low (padding waste) | High (no padding) |
| Throughput | 1× baseline | 5-20× improvement |
| Latency Fairness | Poor (head-of-line blocking) | Good (short requests finish fast) |
| Memory Management | Pre-allocated (wasteful) | Dynamic (PagedAttention) |
| Implementation | Simple | Complex (vLLM, TGI, TensorRT-LLM) |
**Continuous batching is the essential serving optimization for production LLM deployment** — dynamically managing request lifecycles at the iteration level to maximize GPU utilization and throughput, eliminating the idle time waste of static batching and enabling cost-efficient serving of variable-length LLM generation workloads.