load balancing
**Load balancing** is the practice of distributing incoming requests across **multiple servers or instances** to ensure no single server becomes overwhelmed, improving reliability, throughput, and response time. For AI systems, load balancing is critical because LLM inference is resource-intensive and variable in duration.
**Load Balancing Algorithms**
- **Round Robin**: Distribute requests sequentially across servers (1→2→3→1→2→3). Simple but doesn't account for server capacity or current load.
- **Weighted Round Robin**: Assign weights to servers based on capacity — more powerful servers receive more requests.
- **Least Connections**: Route to the server with the fewest active connections. Better for variable-duration requests like LLM inference.
- **Least Response Time**: Route to the server with the lowest current response time.
- **Random**: Select a random server — surprisingly effective and very simple.
- **Consistent Hashing**: Route based on a hash of the request — ensures the same user/query goes to the same server, beneficial for cache locality.
**AI-Specific Load Balancing Considerations**
- **GPU Awareness**: Route requests to servers with available GPU memory — a server with loaded model weights but no GPU memory for inference should not receive new requests.
- **Token-Based Load**: Balance based on **input + output tokens** rather than request count, since a 100-token query consumes far fewer resources than a 10,000-token query.
- **Model Routing**: Route requests to servers hosting the specific model version needed.
- **Priority Queuing**: Route high-priority or paid-tier requests to dedicated, less-loaded servers.
- **Sticky Sessions**: For multi-turn conversations, route all turns to the same server to leverage KV cache reuse.
**Implementation Options**
- **Hardware**: F5, Citrix ADC — enterprise-grade hardware load balancers.
- **Software**: **NGINX**, **HAProxy**, **Envoy** — widely used software load balancers.
- **Cloud**: AWS ALB/NLB, GCP Cloud Load Balancing, Azure Load Balancer — managed cloud services.
- **Service Mesh**: **Istio**, **Linkerd** — provide load balancing as part of service mesh infrastructure.
Load balancing is a **foundational infrastructure component** — production AI systems serving any significant traffic require it for reliability and performance.