continuous batching
**Continuous Batching**
**The Problem with Static Batching**
With static batching, all requests in a batch must complete before new requests can start:
```
Static Batch:
Request 1: [====] (short)
Request 2: [============] (long)
Request 3: [======] (medium)
All must wait for Request 2 to finish.
```
Resources wasted while shorter requests are complete but waiting.
**How Continuous Batching Works**
Process requests as they complete, immediately adding new ones:
```
Continuous Batching:
Request 1: [====]
↳ Request 4: [===]
Request 2: [============]
↳ Request 6: [==]
Request 3: [======]
↳ Request 5: [====]
```
**Iteration-Level Scheduling**
At each decoding iteration:
1. Generate one token for all active requests
2. Check if any request is complete (hit EOS or max tokens)
3. Remove completed requests
4. Add waiting requests from queue (if GPU memory available)
```python
# Pseudocode
while requests_pending:
# Run one forward pass for current batch
for request in active_batch:
new_token = model.generate_one_token(request)
request.append(new_token)
# Remove completed
active_batch = [r for r in active_batch if not r.is_complete()]
# Add new requests
while has_capacity() and waiting_queue:
active_batch.append(waiting_queue.pop())
```
**Benefits**
| Metric | Static Batching | Continuous Batching |
|--------|-----------------|---------------------|
| GPU Utilization | Variable | Consistently high |
| Latency (short requests) | Blocked by long | Minimal waiting |
| Throughput | Lower | 2-3x higher |
| Memory efficiency | Poor | Good (with paging) |
**Implementation in Inference Servers**
| Server | Support |
|--------|---------|
| vLLM | Built-in |
| TGI | Built-in |
| TensorRT-LLM | Built-in |
| Triton + TensorRT | Configurable |
**Configuration Considerations**
**Max Batch Size**
```python
# Limit concurrent requests
max_batch_size = 64 # Adjust based on GPU memory
```
**Preemption**
When memory is tight, may need to preempt (pause) low-priority requests:
```python
preemption_mode = "swap" # swap to CPU, or "recompute"
```
**Queue Management**
- FIFO: First-in, first-out
- Priority: Based on request importance
- Deadline-based: Prioritize requests nearing SLA
Continuous batching is essential for production LLM serving with variable-length requests.