continuous batching

Continuous batching (also called iteration-level batching or in-flight batching) dynamically adds and removes requests from the active batch at each generation step, eliminating the inefficiency of static batching where completed requests block GPU utilization. Problem with static batching: all requests in a batch must complete before any results return—if one request generates 500 tokens and another generates 10, the short request waits idle while the long one finishes, wasting GPU cycles and adding latency. Continuous batching solution: at each decode iteration (token generation step): (1) Generate one token for all active requests; (2) Remove completed requests (hit stop token or max length); (3) Add waiting requests to fill freed slots; (4) Continue to next iteration. Benefits: (1) Higher GPU utilization—freed slots immediately filled with new requests; (2) Lower latency—completed requests return immediately without waiting; (3) Better throughput—no idle GPU cycles from padding or waiting; (4) Predictable performance—steady-state processing rate. Implementation details: (1) KV cache management—must efficiently allocate/deallocate per-request cache; (2) Scheduling—decide which waiting requests to admit based on priority, memory; (3) Prefill scheduling—new request prefill (compute-intensive) interleaved with decode (memory-intensive); (4) Chunked prefill—split long prompt prefill into chunks to avoid blocking decode iterations. Frameworks: (1) vLLM—pioneered PagedAttention + continuous batching; (2) TGI—Hugging Face implementation; (3) TensorRT-LLM—NVIDIA optimized serving; (4) Sarathi-Serve—chunked prefill for balanced scheduling. Performance: continuous batching achieves 2-5× higher throughput than static batching at comparable latency. Industry standard for all production LLM serving deployments.