load balancer
**Load Balancing for ML Services**
**Why Load Balance?**
Distribute traffic across multiple model instances for reliability, scalability, and efficient resource utilization.
**Load Balancing Strategies**
**Round Robin**
Distribute requests evenly:
```nginx
upstream llm_servers {
server llm1.example.com:8000;
server llm2.example.com:8000;
server llm3.example.com:8000;
}
```
**Least Connections**
Route to server with fewest active connections:
```nginx
upstream llm_servers {
least_conn;
server llm1.example.com:8000;
server llm2.example.com:8000;
}
```
**Weighted Distribution**
Allocate based on server capacity:
```nginx
upstream llm_servers {
server gpu-a100.example.com:8000 weight=10;
server gpu-t4.example.com:8000 weight=3;
}
```
**Nginx Configuration**
```nginx
http {
upstream llm_api {
least_conn;
server 10.0.0.1:8000 weight=5;
server 10.0.0.2:8000 weight=5;
# Health checks
keepalive 32;
}
server {
listen 80;
location /api/v1/completions {
proxy_pass http://llm_api;
proxy_http_version 1.1;
proxy_set_header Connection "";
# Timeouts for LLM
proxy_read_timeout 300s;
proxy_connect_timeout 10s;
}
}
}
```
**ML-Specific Considerations**
| Consideration | Solution |
|---------------|----------|
| Long requests | Extended timeouts |
| Streaming | HTTP/1.1, chunked transfer |
| GPU memory | Session affinity if stateful |
| Warm-up | Gradual traffic increase |
**Health Checks**
```nginx
upstream llm_servers {
server llm1:8000;
server llm2:8000;
# Active health check
health_check interval=5s fails=2 passes=1;
}
```
**Session Affinity**
For stateful models (e.g., with KV cache):
```nginx
upstream llm_servers {
ip_hash; # Same IP -> same server
server llm1:8000;
server llm2:8000;
}
```
**Cloud Load Balancers**
| Cloud | Service |
|-------|---------|
| AWS | ALB, NLB |
| GCP | Cloud Load Balancing |
| Azure | Load Balancer |
| Cloudflare | Load Balancing |
**Best Practices**
- Use health checks to remove unhealthy servers
- Set appropriate timeouts for LLM operations
- Consider GPU utilization in routing
- Implement graceful shutdown