Load Balancing for ML Services

Why Load Balance? Distribute traffic across multiple model instances for reliability, scalability, and efficient resource utilization.

Load Balancing Strategies

Round Robin Distribute requests evenly:

upstream llm_servers {
    server llm1.example.com:8000;
    server llm2.example.com:8000;
    server llm3.example.com:8000;
}

Least Connections Route to server with fewest active connections:

upstream llm_servers {
    least_conn;
    server llm1.example.com:8000;
    server llm2.example.com:8000;
}

Weighted Distribution Allocate based on server capacity:

upstream llm_servers {
    server gpu-a100.example.com:8000 weight=10;
    server gpu-t4.example.com:8000 weight=3;
}

Nginx Configuration

http {
    upstream llm_api {
        least_conn;
        server 10.0.0.1:8000 weight=5;
        server 10.0.0.2:8000 weight=5;

        # Health checks
        keepalive 32;
    }

    server {
        listen 80;

        location /api/v1/completions {
            proxy_pass http://llm_api;
            proxy_http_version 1.1;
            proxy_set_header Connection "";

            # Timeouts for LLM
            proxy_read_timeout 300s;
            proxy_connect_timeout 10s;
        }
    }
}

ML-Specific Considerations

Health Checks

upstream llm_servers {
    server llm1:8000;
    server llm2:8000;

    # Active health check
    health_check interval=5s fails=2 passes=1;
}

Session Affinity For stateful models (e.g., with KV cache):

upstream llm_servers {
    ip_hash;  # Same IP -> same server
    server llm1:8000;
    server llm2:8000;
}

Cloud Load Balancers

Best Practices

• Use health checks to remove unhealthy servers
• Set appropriate timeouts for LLM operations
• Consider GPU utilization in routing
• Implement graceful shutdown