Continuous Batching

The Problem with Static Batching With static batching, all requests in a batch must complete before new requests can start:

Static Batch:
Request 1: [====]         (short)
Request 2: [============] (long)
Request 3: [======]       (medium)

All must wait for Request 2 to finish.

Resources wasted while shorter requests are complete but waiting.

How Continuous Batching Works Process requests as they complete, immediately adding new ones:

Continuous Batching:
Request 1: [====]
                ↳ Request 4: [===]
Request 2: [============]
                        ↳ Request 6: [==]
Request 3: [======]
              ↳ Request 5: [====]

Iteration-Level Scheduling At each decoding iteration: 1. Generate one token for all active requests 2. Check if any request is complete (hit EOS or max tokens) 3. Remove completed requests 4. Add waiting requests from queue (if GPU memory available)

# Pseudocode
while requests_pending:
    # Run one forward pass for current batch
    for request in active_batch:
        new_token = model.generate_one_token(request)
        request.append(new_token)

    # Remove completed
    active_batch = [r for r in active_batch if not r.is_complete()]

    # Add new requests
    while has_capacity() and waiting_queue:
        active_batch.append(waiting_queue.pop())

Benefits

Implementation in Inference Servers

Configuration Considerations

Max Batch Size

# Limit concurrent requests
max_batch_size = 64  # Adjust based on GPU memory

Preemption When memory is tight, may need to preempt (pause) low-priority requests:

preemption_mode = "swap"  # swap to CPU, or "recompute"

Queue Management

• FIFO: First-in, first-out
• Priority: Based on request importance
• Deadline-based: Prioritize requests nearing SLA

Continuous batching is essential for production LLM serving with variable-length requests.