system design
**System design for LLM applications** involves **architecting scalable, reliable infrastructure to serve AI capabilities to users** — addressing unique challenges like variable latency, high memory requirements, and non-deterministic outputs while applying traditional system design principles for load balancing, caching, and fault tolerance.
**What Is LLM System Design?**
- **Definition**: Architecture for production LLM serving at scale.
- **Challenges**: High latency, GPU costs, variable load.
- **Goals**: Reliability, performance, cost efficiency.
- **Approach**: Adapt traditional patterns for AI constraints.
**Why LLM System Design Differs**
- **Resource Intensive**: Single request may use 24GB+ GPU memory.
- **Variable Latency**: Responses take 100ms to 30s depending on length.
- **Stateful Conversations**: Context must be maintained across requests.
- **Non-Deterministic**: Same input can produce different outputs.
- **Expensive Operations**: Each token costs money.
**High-Level Architecture**
```svg
```
**Key Components**
**API Layer**:
```python
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
import asyncio
app = FastAPI()
@app.post("/v1/chat/completions")
async def chat_completion(request: ChatRequest):
# Validate
if len(request.messages) == 0:
raise HTTPException(400, "Messages required")
# Check cache
cache_key = hash_request(request)
if cached := await cache.get(cache_key):
return cached
# Route to appropriate model
model_endpoint = get_model_endpoint(request.model)
# Generate (with timeout)
try:
response = await asyncio.wait_for(
generate(model_endpoint, request),
timeout=60.0
)
except asyncio.TimeoutError:
raise HTTPException(504, "Generation timeout")
# Cache and return
await cache.set(cache_key, response, ttl=3600)
return response
```
**Scaling Strategies**
**Horizontal Scaling**:
```
Load Pattern | Strategy
----------------------|----------------------------------
Bursty traffic | Auto-scaling GPU instances
Predictable peaks | Scheduled scaling
Global users | Multi-region deployment
Cost optimization | Spot instances + fallback
```
**Caching Layers**:
```
Layer | Cache What | TTL
-------------------|----------------------|----------
Response cache | Full responses | 1-24 hours
Embedding cache | Vector embeddings | Days
KV cache | Attention states | Session
Prefix cache | System prompts | Hours
```
**Multi-Model Routing**
```python
def route_request(request):
"""Route to appropriate model based on complexity."""
# Simple queries → small/fast model
if is_simple_query(request):
return "gpt-4o-mini"
# Complex reasoning → large model
if needs_complex_reasoning(request):
return "gpt-4o"
# Default
return "gpt-4o-mini"
def is_simple_query(request):
prompt = request.messages[-1].content
return (
len(prompt) < 100 and
not any(word in prompt for word in
["explain", "analyze", "compare"])
)
```
**Reliability Patterns**
**Circuit Breaker**:
```python
class CircuitBreaker:
def __init__(self, failure_threshold=5, reset_timeout=60):
self.failures = 0
self.state = "closed"
self.last_failure = None
async def call(self, func, *args):
if self.state == "open":
if time.time() - self.last_failure > self.reset_timeout:
self.state = "half-open"
else:
raise CircuitOpenError()
try:
result = await func(*args)
self.failures = 0
self.state = "closed"
return result
except Exception as e:
self.failures += 1
self.last_failure = time.time()
if self.failures >= self.failure_threshold:
self.state = "open"
raise
```
**Fallback Chain**:
```python
async def generate_with_fallback(request):
providers = ["openai", "anthropic", "local"]
for provider in providers:
try:
return await generate(provider, request)
except Exception as e:
logger.warning(f"{provider} failed: {e}")
continue
raise AllProvidersFailedError()
```
**Monitoring & Observability**
```
Metric | What to Track
--------------------|--------------------------------
Latency (P50/P95) | TTFT, total generation time
Throughput | Requests/sec, tokens/sec
Error rate | 4xx, 5xx, timeouts
GPU utilization | Memory, compute usage
Cost | Tokens per request, $/query
```
System design for LLM applications requires **balancing performance, reliability, and cost** — applying proven distributed systems patterns while adapting to the unique constraints of GPU-bound, high-latency inference workloads.