what is continuous batching
**Continuous batching is the serving-system technique that decides which requests actually get processed together on any given decode step, and it's the layer where every optimization covered so far in this series meets real, messy traffic.** Naive batching waits for a group of requests to arrive, runs them together as a fixed batch, and only starts the next batch once every request in the current one has finished — which wastes enormous capacity, because different requests generate wildly different numbers of tokens, so a batch sits idle waiting for its slowest member to finish while shorter requests have long since completed. Continuous batching (sometimes called in-flight or dynamic batching) fixes this by making the batch itself fluid: as soon as any request finishes, a new waiting request is slotted into its place immediately, on the very next decode step, without waiting for the whole batch to drain.
**This matters specifically because decode is memory-bandwidth-bound, and batching is the main lever for actually using bandwidth efficiently rather than leaving it idle.** The earlier entry on LLM inference established that decode has to re-read the model's weights and KV cache on every single step, largely independent of how many requests are being processed at once — which means processing several requests' decode steps together shares that expensive memory read across all of them, dramatically improving how many tokens-per-second of useful work come out of every byte moved. A batch that's allowed to shrink and refill continuously keeps that shared memory read as full and productive as possible at every single step, instead of degrading toward one request at a time as a fixed batch drains.
```svg
```
**Continuous batching is also where every earlier optimization in this series stops being independent and starts interacting.** Quantization and grouped/multi-query attention shrink how much memory traffic each request's decode step costs, which means more requests can be packed into the same batch before hitting a memory-bandwidth ceiling. Speculative decoding complicates batching because different requests in the batch may accept different numbers of draft tokens on any given step, so a serving system has to handle a batch where members advance by different amounts simultaneously. Mixture-of-experts complicates it further still, because different tokens in the same batch may route to different experts, meaning the batch isn't just sharing one memory read anymore — it's coordinating several different experts' memory reads across the same step. None of these techniques were designed in isolation from serving; continuous batching is the scheduling layer that has to reconcile all of them at once, in real time, request by request.
| Batching Approach | New Requests Join | Idle Time on Early Finishers | Interacts With |
|---|---|---|---|
| Static Batching | Only after the whole batch fully drains | High — batch runs at the pace of its slowest member | Simple to implement, poor hardware utilization |
| Continuous Batching | Immediately, on the very next decode step | Minimal — slots refill as soon as they open | Quantization, GQA/MQA, speculative decoding, MoE routing |
```flowchart
st=>start: Decode step begins for the current batch
check=>operation: Check each request in the batch for completion
finish=>operation: Any finished request is removed from the batch immediately
refill=>operation: Next waiting request from the queue is slotted into the newly opened space
runstep=>operation: One shared decode step runs across the full, refilled batch
loop=>operation: Repeat for every subsequent decode step
pass=>end: Sustained high throughput with minimal idle batch capacity
st->check->finish->refill->runstep->loop->pass
```
**For AI accelerator and system design, continuous batching is the reason serving throughput numbers depend as much on the scheduler as on the chip itself.** A chip with excellent memory bandwidth and every architectural optimization covered in this series still underdelivers if the serving software feeding it lets batch slots sit idle; conversely, a well-implemented continuous batching scheduler can let a modestly specified chip approach the sustained throughput of a much larger one simply by never wasting a decode step. That's why production LLM-serving stacks treat the batching scheduler as a first-class piece of the system, on equal footing with the accelerator hardware and the model architecture choices — quantization, attention variant, speculative decoding, and MoE routing all ultimately funnel through this one scheduling decision, made fresh on every single decode step.