Batching

Batching is the practice of processing several inputs together in one pass through a model rather than one at a time, so that the fixed cost of loading weights and launching computation is shared across many requests. Accelerators are built for large, regular matrix multiplications, and a single request often leaves most of their arithmetic units idle; stacking requests into a batch turns many small matrix-vector operations into fewer, larger matrix-matrix operations that use the hardware far more efficiently. The word covers two distinct settings that share an idea: in training, a batch is the group of examples whose gradients are averaged for one optimizer step, and in inference, a batch is the group of requests served together by the same forward pass.

In serving, the central trade is throughput against latency: bigger batches raise total tokens per second and lower cost per token, but each individual request waits longer. Model weights must be read from memory for every forward pass regardless of how many sequences ride along, and during autoregressive decoding that memory traffic, not arithmetic, is usually the limit. Adding a sequence to the batch therefore costs little extra time per step while producing an extra token, which is why batching is the main lever for cost-efficient inference. The price is queueing delay while a batch fills, slower individual steps as the batch grows, and a larger kv-cache footprint, since every sequence in the batch holds its own cached keys and values. Interactive chat applications tune for low time to first token and smooth streaming; offline jobs such as bulk summarization or evaluation can use very large batches and care only about total throughput.

Static batching, the classic approach, groups requests, pads them to a common length and runs them until the whole group finishes, which wastes capacity badly for text generation. Requests produce outputs of very different lengths, so a batch that includes one long answer keeps finished sequences occupying slots, generating nothing useful, until the slowest one is done, and new arrivals must wait for the next batch. Padding short prompts to match long ones adds further wasted computation. Dynamic batching, offered for example by NVIDIA's Triton Inference Server, improves on this by waiting a short, configurable time to assemble a batch from requests as they arrive, which suits models with a single fixed-length output such as classifiers or image models, but it still treats the batch as an indivisible unit.

Continuous batching, introduced for transformer generation in the Orca system (Yu et al., OSDI 2022) under the name iteration-level scheduling, schedules at the granularity of one decoding step instead of one whole request. After every iteration the scheduler can remove sequences that have finished and admit waiting ones immediately, so slots never sit idle and short requests do not have to wait behind long ones. Orca reported large throughput gains over earlier serving systems at comparable latency. The idea is now standard in open serving engines, including vLLM (Kwon et al., 2023), whose PagedAttention manages the kv-cache in fixed-size blocks like virtual memory pages so that sequences of unpredictable length can be packed without fragmentation, and in Hugging Face Text Generation Inference and NVIDIA TensorRT-LLM, where it is sometimes called in-flight batching.

Prefill and decode behave differently, which complicates batching and has driven recent scheduler design. Processing the prompt (prefill) is compute-heavy and handles many tokens in parallel, while generating output (decode) is memory-bound and emits one token per sequence per step. A long prefill dropped into a running batch can stall everyone else's decoding and cause visible hiccups in streaming. Techniques such as chunked prefill, which splits a long prompt across several iterations, and disaggregated serving, which runs prefill and decode on separate groups of machines, exist to smooth this out. Speculative decoding and prefix caching interact with batching as well, since shared prompt prefixes can be computed once and reused across requests.

Several misconceptions are common. A larger batch is not always faster for the user: past the point where the hardware is saturated, throughput stops improving while latency keeps climbing. Training batch size has its own separate consequences, since very large batches can hurt generalization or require learning-rate scaling and warmup, as studied in work such as Goyal et al. (2017) on training ImageNet in an hour with large minibatches. Batch-dependent layers such as batch-normalization behave differently at training and inference, so inference batching must not change a model's outputs; in practice small numerical differences can still appear because batch composition alters the order of floating-point operations, which is one reason identical requests are not always bit-for-bit reproducible even at temperature zero.

Read batching as the economics of sharing: an accelerator's expensive trip to memory is paid once per step, and continuous batching is the discipline of keeping every seat on that trip filled while keeping each rider's wait acceptable.