timeout
**Timeouts and SLAs for LLM Services**
**Why Timeouts Matter**
LLM operations can be slow. Without proper timeouts, resources get exhausted and user experience suffers.
**Timeout Layers**
```
Client timeout (shortest)
|
v
API Gateway timeout
|
v
Application timeout
|
v
LLM Provider timeout (longest)
```
**Setting Timeouts**
```python
import httpx
import asyncio
async def call_with_timeout(prompt, timeout=30):
async with httpx.AsyncClient(timeout=timeout) as client:
try:
response = await client.post(
"https://api.openai.com/v1/completions",
json={"prompt": prompt},
timeout=timeout
)
return response.json()
except httpx.TimeoutException:
raise TimeoutError(f"LLM call exceeded {timeout}s")
```
**Deadline Propagation**
Pass remaining time through the stack:
```python
def process_with_deadline(request):
deadline = time.time() + 30 # 30 second deadline
# Pass remaining time to each step
remaining = deadline - time.time()
result1 = step1(request, timeout=remaining * 0.3)
remaining = deadline - time.time()
result2 = step2(result1, timeout=remaining * 0.5)
remaining = deadline - time.time()
return step3(result2, timeout=remaining)
```
**SLA Levels**
| Tier | P50 | P99 | Timeout |
|------|-----|-----|---------|
| Fast | 1s | 5s | 10s |
| Standard | 5s | 30s | 60s |
| Batch | 30s | 120s | 300s |
**Monitoring SLA Compliance**
```python
import time
from prometheus_client import Histogram
latency = Histogram(
"llm_request_latency_seconds",
"LLM request latency",
buckets=[0.5, 1, 2, 5, 10, 30, 60]
)
@latency.time()
def call_llm(prompt):
return llm.generate(prompt)
# Alert on SLA breach
def check_sla(latencies, sla_p99=30):
p99 = percentile(latencies, 99)
if p99 > sla_p99:
alert("P99 latency exceeds SLA!")
```
**Timeout Strategies**
| Strategy | Description |
|----------|-------------|
| Hard timeout | Cancel after limit |
| Soft timeout | Return partial result |
| Adaptive | Adjust based on load |
| Budget-based | Split time across steps |
**Handling Timeout**
```python
async def call_with_retry_on_timeout(prompt, max_attempts=2):
for attempt in range(max_attempts):
try:
return await call_with_timeout(prompt, timeout=30)
except TimeoutError:
if attempt == max_attempts - 1:
# Return cached or fallback
return get_fallback_response(prompt)
await asyncio.sleep(1)
```
**Best Practices**
- Set timeouts at every layer
- Make timeouts configurable
- Log timeout occurrences
- Monitor and alert on SLA breaches
- Consider tail latency (P99, P999)