batch size optimization

Batch size optimization tunes the number of concurrent requests processed together during LLM inference to maximize throughput while meeting latency requirements, balancing GPU utilization against response time. Key tradeoff: (1) Small batch—low latency per request but underutilizes GPU compute (especially during decode phase); (2) Large batch—high throughput and GPU utilization but increased per-request latency and memory pressure. Batch size constraints: (1) GPU memory—each request requires KV cache storage (grows with sequence length), limiting maximum batch size; (2) Latency SLO—maximum acceptable time-to-first-token and inter-token delay; (3) Compute saturation—point where adding more requests doesn't increase tokens/second. Memory calculation: KV cache per request = 2 × n_layers × n_heads × head_dim × seq_len × dtype_bytes. For 70B model with 4K context in FP16: ~10GB per request, limiting H100 (80GB) to ~4-5 concurrent requests (after model weights). Optimization strategies: (1) Quantized KV cache—INT8 or FP8 cache doubles batch capacity; (2) Multi-query attention (MQA)/grouped-query attention (GQA)—reduces KV cache size 8-32×; (3) PagedAttention—eliminates memory fragmentation, maximizes usable memory; (4) Dynamic batching—adjust batch size based on current load and request characteristics; (5) Prefix caching—share KV cache for common prompt prefixes. Profiling approach: sweep batch sizes, measure throughput (tokens/s) and latency (P50/P99), find knee of curve where throughput plateaus before latency degrades. Different batch sizes for prefill vs. decode: chunked prefill processes long inputs in smaller chunks to avoid blocking decode of other requests. Optimal batch size is workload-dependent—varies with model size, sequence length distribution, hardware, and latency requirements.