System design for LLM applications involves architecting scalable, reliable infrastructure to serve AI capabilities to users — addressing unique challenges like variable latency, high memory requirements, and non-deterministic outputs while applying traditional system design principles for load balancing, caching, and fault tolerance.
What Is LLM System Design?
- Definition: Architecture for production LLM serving at scale.
- Challenges: High latency, GPU costs, variable load.
- Goals: Reliability, performance, cost efficiency.
- Approach: Adapt traditional patterns for AI constraints.
Why LLM System Design Differs
- Resource Intensive: Single request may use 24GB+ GPU memory.
- Variable Latency: Responses take 100ms to 30s depending on length.
- Stateful Conversations: Context must be maintained across requests.
- Non-Deterministic: Same input can produce different outputs.
- Expensive Operations: Each token costs money.
High-Level Architecture
<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
<rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
<text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">System Design — Scalable Architecture Patterns</text>
<text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">decompose, scale horizontally, cache aggressively, queue for resilience, shard for growth</text>
<!-- === TOP: Typical web-scale architecture === -->
<rect x="25" y="62" width="710" height="155" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
<text x="380" y="82" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Scalable Web Architecture</text>
<!-- Client -->
<rect x="40" y="95" width="55" height="28" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
<text x="67" y="112" fill="#6ee7b7" font-size="8" text-anchor="middle">Clients</text>
<path d="M98,109 L118,109" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
<polygon points="116,106 122,109 116,112" fill="#8b98a5"/>
<!-- CDN -->
<rect x="125" y="95" width="50" height="28" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.8"/>
<text x="150" y="112" fill="#93c5fd" font-size="8" text-anchor="middle">CDN</text>
<path d="M178,109 L198,109" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
<polygon points="196,106 202,109 196,112" fill="#8b98a5"/>
<!-- Load Balancer -->
<rect x="205" y="90" width="65" height="38" rx="4" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.9"/>
<text x="237" y="108" fill="#fbbf24" font-size="8" text-anchor="middle">Load</text>
<text x="237" y="120" fill="#fbbf24" font-size="8" text-anchor="middle">Balancer</text>
<!-- App servers (multiple) -->
<path d="M273,100 L310,95" fill="none" stroke="#8b98a5" stroke-width="0.6"/>
<path d="M273,109 L310,109" fill="none" stroke="#8b98a5" stroke-width="0.6"/>
<path d="M273,118 L310,123" fill="none" stroke="#8b98a5" stroke-width="0.6"/>
<rect x="313" y="86" width="55" height="20" rx="2" fill="#1a1520" stroke="#a78bfa" stroke-width="0.7"/>
<text x="340" y="99" fill="#c4b5fd" font-size="7" text-anchor="middle">App Server</text>
<rect x="313" y="108" width="55" height="20" rx="2" fill="#1a1520" stroke="#a78bfa" stroke-width="0.7"/>
<text x="340" y="121" fill="#c4b5fd" font-size="7" text-anchor="middle">App Server</text>
<rect x="313" y="130" width="55" height="20" rx="2" fill="#1a1520" stroke="#a78bfa" stroke-width="0.7"/>
<text x="340" y="143" fill="#c4b5fd" font-size="7" text-anchor="middle">App Server</text>
<!-- Cache -->
<path d="M370,109 L395,95" fill="none" stroke="#8b98a5" stroke-width="0.6"/>
<rect x="398" y="82" width="55" height="28" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
<text x="425" y="99" fill="#6ee7b7" font-size="8" text-anchor="middle">Cache</text>
<text x="425" y="109" fill="#6b7684" font-size="6.5" text-anchor="middle">Redis/Memcached</text>
<!-- Database -->
<path d="M370,120 L395,130" fill="none" stroke="#8b98a5" stroke-width="0.6"/>
<rect x="398" y="118" width="55" height="38" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.8"/>
<text x="425" y="135" fill="#93c5fd" font-size="8" text-anchor="middle">DB</text>
<text x="425" y="147" fill="#6b7684" font-size="6.5" text-anchor="middle">Primary</text>
<!-- Read replicas -->
<path d="M456,137 L480,130" fill="none" stroke="#8b98a5" stroke-width="0.5"/>
<path d="M456,137 L480,145" fill="none" stroke="#8b98a5" stroke-width="0.5"/>
<rect x="483" y="120" width="50" height="18" rx="2" fill="#0b1220" stroke="#233043" stroke-width="0.6"/>
<text x="508" y="132" fill="#8b98a5" font-size="6.5" text-anchor="middle">Replica</text>
<rect x="483" y="140" width="50" height="18" rx="2" fill="#0b1220" stroke="#233043" stroke-width="0.6"/>
<text x="508" y="152" fill="#8b98a5" font-size="6.5" text-anchor="middle">Replica</text>
<!-- Message queue -->
<path d="M370,135 L395,170" fill="none" stroke="#8b98a5" stroke-width="0.6"/>
<rect x="398" y="162" width="55" height="28" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
<text x="425" y="177" fill="#fbbf24" font-size="7.5" text-anchor="middle">Queue</text>
<text x="425" y="188" fill="#6b7684" font-size="6.5" text-anchor="middle">Kafka/SQS</text>
<!-- Workers -->
<path d="M456,176 L485,176" fill="none" stroke="#8b98a5" stroke-width="0.5"/>
<rect x="488" y="164" width="50" height="24" rx="2" fill="#1a1520" stroke="#a78bfa" stroke-width="0.7"/>
<text x="513" y="179" fill="#c4b5fd" font-size="7" text-anchor="middle">Workers</text>
<!-- Object storage -->
<rect x="570" y="95" width="60" height="28" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
<text x="600" y="112" fill="#6ee7b7" font-size="7.5" text-anchor="middle">Blob/S3</text>
<!-- Search -->
<rect x="570" y="135" width="60" height="28" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.8"/>
<text x="600" y="152" fill="#93c5fd" font-size="7.5" text-anchor="middle">Search</text>
<!-- Monitoring -->
<rect x="645" y="95" width="70" height="28" rx="3" fill="#1a0f0f" stroke="#f87171" stroke-width="0.8"/>
<text x="680" y="112" fill="#f87171" font-size="7.5" text-anchor="middle">Monitoring</text>
<!-- === MIDDLE LEFT: Core patterns === -->
<rect x="25" y="225" width="350" height="108" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
<text x="200" y="243" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Core Scaling Patterns</text>
<text x="45" y="263" fill="#34d399" font-size="8.5" font-weight="600">Horizontal scaling</text>
<text x="170" y="263" fill="#8b98a5" font-size="8.5">add more machines (stateless)</text>
<text x="45" y="279" fill="#60a5fa" font-size="8.5" font-weight="600">Caching</text>
<text x="105" y="279" fill="#8b98a5" font-size="8.5">L1 local → L2 Redis → DB (reduce reads 90%)</text>
<text x="45" y="295" fill="#fbbf24" font-size="8.5" font-weight="600">Sharding</text>
<text x="110" y="295" fill="#8b98a5" font-size="8.5">partition data by key (range / hash / geo)</text>
<text x="45" y="311" fill="#c4b5fd" font-size="8.5" font-weight="600">Async processing</text>
<text x="165" y="311" fill="#8b98a5" font-size="8.5">queue heavy work, respond immediately</text>
<text x="45" y="327" fill="#f87171" font-size="8.5" font-weight="600">Event sourcing</text>
<text x="145" y="327" fill="#8b98a5" font-size="8.5">append-only log, rebuild state from events</text>
<!-- === MIDDLE RIGHT: CAP theorem === -->
<rect x="390" y="225" width="345" height="108" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
<text x="562" y="243" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Tradeoffs (CAP / PACELC)</text>
<text x="410" y="263" fill="#60a5fa" font-size="8.5" font-weight="600">C (Consistency):</text>
<text x="530" y="263" fill="#8b98a5" font-size="8.5">every read returns latest write</text>
<text x="410" y="279" fill="#34d399" font-size="8.5" font-weight="600">A (Availability):</text>
<text x="520" y="279" fill="#8b98a5" font-size="8.5">every request gets a response</text>
<text x="410" y="295" fill="#fbbf24" font-size="8.5" font-weight="600">P (Partition tolerance):</text>
<text x="560" y="295" fill="#8b98a5" font-size="8.5">survives network splits</text>
<text x="410" y="317" fill="#f87171" font-size="8.5">Pick 2 of 3 during partition. In practice: choose</text>
<text x="410" y="331" fill="#f87171" font-size="8.5">between consistency (CP) or availability (AP).</text>
<!-- === BOTTOM: Numbers to know === -->
<rect x="25" y="343" width="710" height="62" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
<text x="380" y="361" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Latency Numbers Every Engineer Should Know</text>
<text x="45" y="381" fill="#34d399" font-size="8.5">L1 cache: 0.5ns</text>
<text x="155" y="381" fill="#60a5fa" font-size="8.5">RAM: 100ns</text>
<text x="255" y="381" fill="#fbbf24" font-size="8.5">SSD: 150μs</text>
<text x="345" y="381" fill="#c4b5fd" font-size="8.5">HDD: 10ms</text>
<text x="440" y="381" fill="#f87171" font-size="8.5">Network (DC): 0.5ms</text>
<text x="600" y="381" fill="#8b98a5" font-size="8.5">Cross-region: 50-100ms</text>
<text x="45" y="397" fill="#8b98a5" font-size="8">1M QPS × 1ms/req = 1000 servers | Add cache → 100 servers | Add CDN → 20 servers</text>
<!-- Key insight -->
<rect x="25" y="413" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
<text x="380" y="428" fill="#fbbf24" font-size="9" text-anchor="middle">The system design process: requirements → estimates → high-level design → deep dive → bottlenecks → tradeoffs.</text>
<text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Scalable systems are built from simple, composable pieces: stateless services, caches, queues, and sharded storage.</text>
</svg>
Key Components
API Layer:
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
import asyncio
app = FastAPI()
@app.post("/v1/chat/completions")
async def chat_completion(request: ChatRequest):
# Validate
if len(request.messages) == 0:
raise HTTPException(400, "Messages required")
# Check cache
cache_key = hash_request(request)
if cached := await cache.get(cache_key):
return cached
# Route to appropriate model
model_endpoint = get_model_endpoint(request.model)
# Generate (with timeout)
try:
response = await asyncio.wait_for(
generate(model_endpoint, request),
timeout=60.0
)
except asyncio.TimeoutError:
raise HTTPException(504, "Generation timeout")
# Cache and return
await cache.set(cache_key, response, ttl=3600)
return response
Scaling Strategies
Horizontal Scaling:
Load Pattern | Strategy
----------------------|----------------------------------
Bursty traffic | Auto-scaling GPU instances
Predictable peaks | Scheduled scaling
Global users | Multi-region deployment
Cost optimization | Spot instances + fallback
Caching Layers:
Layer | Cache What | TTL
-------------------|----------------------|----------
Response cache | Full responses | 1-24 hours
Embedding cache | Vector embeddings | Days
KV cache | Attention states | Session
Prefix cache | System prompts | Hours
Multi-Model Routing
def route_request(request):
"""Route to appropriate model based on complexity."""
# Simple queries → small/fast model
if is_simple_query(request):
return "gpt-4o-mini"
# Complex reasoning → large model
if needs_complex_reasoning(request):
return "gpt-4o"
# Default
return "gpt-4o-mini"
def is_simple_query(request):
prompt = request.messages[-1].content
return (
len(prompt) < 100 and
not any(word in prompt for word in
["explain", "analyze", "compare"])
)
Reliability Patterns
Circuit Breaker:
class CircuitBreaker:
def __init__(self, failure_threshold=5, reset_timeout=60):
self.failures = 0
self.state = "closed"
self.last_failure = None
async def call(self, func, *args):
if self.state == "open":
if time.time() - self.last_failure > self.reset_timeout:
self.state = "half-open"
else:
raise CircuitOpenError()
try:
result = await func(*args)
self.failures = 0
self.state = "closed"
return result
except Exception as e:
self.failures += 1
self.last_failure = time.time()
if self.failures >= self.failure_threshold:
self.state = "open"
raise
Fallback Chain:
async def generate_with_fallback(request):
providers = ["openai", "anthropic", "local"]
for provider in providers:
try:
return await generate(provider, request)
except Exception as e:
logger.warning(f"{provider} failed: {e}")
continue
raise AllProvidersFailedError()
Monitoring & Observability
Metric | What to Track
--------------------|--------------------------------
Latency (P50/P95) | TTFT, total generation time
Throughput | Requests/sec, tokens/sec
Error rate | 4xx, 5xx, timeouts
GPU utilization | Memory, compute usage
Cost | Tokens per request, $/query
System design for LLM applications requires balancing performance, reliability, and cost — applying proven distributed systems patterns while adapting to the unique constraints of GPU-bound, high-latency inference workloads.
system designarchitecturescalingload balancingcachingreliabilityllm infrastructure
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.