request batching

Request batching groups multiple independent inference requests together for simultaneous GPU processing, amortizing the overhead of model weight loading and improving hardware utilization. Why batch: during LLM decode, each token generation requires reading all model weights from memory—with a single request, GPU compute units are idle while waiting for memory. Batching multiple requests reuses the same weight reads across all requests, converting memory-bound to compute-bound operation. Batching types: (1) Static batching—collect fixed number of requests, process together, wait for all to complete before returning any results (simple but wasteful); (2) Dynamic batching—wait for short timeout to collect requests, process available batch (better latency-throughput balance); (3) Continuous batching—requests join and leave batch dynamically as they arrive and complete (optimal utilization). Static batching inefficiency: requests with different output lengths complete at different times—short requests wait for longest request, wasting GPU cycles and increasing latency. Token-level batching: in autoregressive generation, batch at each token step—completed requests leave, new requests join. This is the foundation of continuous batching. Implementation considerations: (1) Padding—different input lengths require padding or variable-length handling; (2) Memory management—KV cache allocation per request; (3) Priority handling—some requests may have higher SLO requirements; (4) Preemption—ability to pause low-priority requests when high-priority arrives. Frameworks: vLLM, TGI, TensorRT-LLM, Triton Inference Server all implement advanced batching. Throughput improvement: batching can improve throughput 5-20× compared to single-request processing on the same hardware. Request batching is the most fundamental optimization for LLM serving cost efficiency and is implemented in every production serving system.