continuous batching inference

**Continuous Batching** is **the inference serving technique that dynamically adds and removes sequences from batches at each generation step rather than waiting for all sequences to complete** — improving GPU utilization by 2-10× and reducing average latency by 30-50% compared to static batching, enabling high-throughput LLM serving systems like vLLM and TensorRT-LLM to serve 10-100× more requests per GPU. **Static Batching Limitations:** - **Batch Completion Wait**: static batching processes fixed batch of sequences; waits for longest sequence to complete; short sequences finish early but GPU idles; wasted computation - **Length Variation**: real-world requests have 10-100× length variation (10 tokens to 1000+ tokens); batch completion time determined by longest sequence; average utilization 20-40% - **Example**: batch of 32 sequences, 31 complete in 50 tokens, 1 requires 500 tokens; GPU idles for 31 sequences while processing last sequence; 97% waste - **Throughput Impact**: low utilization directly reduces throughput; serving 100 requests/sec with 40% utilization could serve 250 requests/sec at 100% utilization **Continuous Batching Algorithm:** - **Iteration-Level Batching**: form new batch at each generation step; add newly arrived requests; remove completed sequences; batch size varies dynamically - **Sequence Lifecycle**: request arrives → added to batch at next step → generates tokens → completes → removed from batch; no waiting for batch completion - **Memory Management**: allocate memory for each sequence independently; deallocate when sequence completes; no memory waste from completed sequences - **Scheduling**: priority queue of waiting requests; add highest-priority requests to batch when space available; fair scheduling or priority-based **Implementation Details:** - **KV Cache Management**: each sequence has independent KV cache; caches grow/shrink as sequences added/removed; requires dynamic memory allocation - **Attention Masking**: variable-length sequences in batch require attention masks; each sequence attends only to its own tokens; padding not needed - **Batch Size Limits**: maximum batch size limited by memory (KV cache + activations); dynamically adjust based on sequence lengths; longer sequences reduce max batch size - **Prefill vs Decode**: prefill (first token) processes full prompt; decode (subsequent tokens) processes one token; separate batching for prefill and decode improves efficiency **Performance Improvements:** - **GPU Utilization**: increases from 20-40% (static) to 60-80% (continuous); 2-4× improvement; directly translates to throughput increase - **Throughput**: 2-10× higher requests/second depending on length distribution; larger improvement for higher length variation; typical 3-5× in production - **Latency**: reduces average latency by 30-50%; short sequences don't wait for long sequences; improves user experience; critical for interactive applications - **Cost Efficiency**: 3-5× more requests per GPU; reduces infrastructure cost by 60-80%; major cost savings for large-scale deployment **Memory Management:** - **PagedAttention**: treats KV cache like virtual memory; allocates in fixed-size blocks (pages); enables efficient memory utilization; used in vLLM - **Block Allocation**: allocate blocks on-demand as sequence grows; deallocate when sequence completes; eliminates fragmentation; achieves 90-95% memory utilization - **Copy-on-Write**: sequences with shared prefix (e.g., system prompt) share KV cache blocks; only copy when sequences diverge; critical for multi-turn conversations - **Memory Limits**: maximum concurrent sequences limited by total KV cache memory; dynamically adjust based on sequence lengths; reject requests when memory full **Scheduling Strategies:** - **FCFS (First-Come-First-Served)**: simple fair scheduling; add requests in arrival order; easy to implement; may starve long requests - **Shortest-Job-First**: prioritize requests with shorter expected length; minimizes average latency; requires length prediction; may starve long requests - **Priority-Based**: assign priorities to requests; serve high-priority first; useful for multi-tenant systems; requires priority mechanism - **Fair Scheduling**: ensure all requests make progress; prevent starvation; balance throughput and fairness; used in production systems **Prefill-Decode Separation:** - **Prefill Batching**: batch multiple prefill requests together; process full prompts in parallel; high memory usage (full prompt activations); limited batch size - **Decode Batching**: batch decode steps from multiple sequences; process one token per sequence; low memory usage; large batch sizes possible - **Separate Queues**: maintain separate queues for prefill and decode; schedule independently; optimize for different characteristics; improves overall efficiency - **Chunked Prefill**: split long prompts into chunks; process chunks like decode steps; reduces memory spikes; enables larger prefill batches **Framework Implementations:** - **vLLM**: pioneering continuous batching implementation; PagedAttention for memory management; achieves 10-20× throughput vs naive serving; open-source, production-ready - **TensorRT-LLM**: NVIDIA's inference framework; continuous batching with optimized CUDA kernels; in-flight batching; highest performance on NVIDIA GPUs - **Text Generation Inference (TGI)**: Hugging Face's serving framework; continuous batching support; easy deployment; good for diverse models - **Ray Serve**: distributed serving with continuous batching; scales to multiple nodes; good for large-scale deployment; integrates with Ray ecosystem **Production Deployment:** - **Request Routing**: load balancer distributes requests across replicas; each replica runs continuous batching; scales horizontally; handles high request rates - **Monitoring**: track batch size, utilization, latency, throughput; identify bottlenecks; adjust configuration; critical for optimization - **Auto-Scaling**: scale replicas based on request rate and latency; continuous batching improves utilization, reduces scaling needs; cost savings - **Fault Tolerance**: handle failures gracefully; retry failed requests; checkpoint long-running sequences; critical for production reliability **Advanced Techniques:** - **Speculative Decoding Integration**: combine continuous batching with speculative decoding; multiplicative speedup; 5-10× total improvement vs naive serving - **Multi-LoRA Serving**: serve multiple LoRA adapters in same batch; different adapter per sequence; enables multi-tenant serving; critical for customization - **Quantization**: INT8/INT4 quantization reduces memory; enables larger batches; combined with continuous batching for maximum throughput - **Prefix Caching**: cache KV for common prefixes (system prompts); share across requests; reduces computation; improves throughput for repetitive prompts **Use Cases:** - **Chatbots**: high request rate, variable response length; continuous batching critical for cost-effective serving; 3-5× cost reduction typical - **Code Completion**: short prompts, variable completion length; benefits from continuous batching; improves latency and throughput - **Content Generation**: variable-length outputs (summaries, articles); continuous batching prevents long generations from blocking short ones - **API Serving**: diverse request patterns; continuous batching handles variation efficiently; critical for production API endpoints **Best Practices:** - **Batch Size**: set maximum batch size based on memory; monitor actual batch size; adjust based on request patterns; typical max 32-128 sequences - **Timeout**: set generation timeout to prevent runaway sequences; release resources from timed-out sequences; critical for stability - **Memory Reservation**: reserve memory for incoming requests; prevents out-of-memory errors; maintain headroom for request spikes - **Profiling**: profile end-to-end latency; identify bottlenecks (prefill, decode, scheduling); optimize based on measurements Continuous Batching is **the technique that transformed LLM serving economics** — by eliminating the waste of static batching and dynamically managing sequences, it achieves 2-10× higher throughput and 30-50% lower latency, making large-scale LLM deployment practical and cost-effective for production applications.