LLM serving and APIs are the infrastructure and interfaces that deploy AI models as production services — wrapping trained models in scalable API endpoints with authentication, rate limiting, streaming, and monitoring, enabling applications from chatbots to coding assistants to integrate AI capabilities reliably.

What Is LLM Serving?

• Definition: Deploying trained LLMs as accessible API services.
• Components: Inference engine, API gateway, load balancing, monitoring.
• Interface: REST or gRPC endpoints for text generation.
• Challenge: Scale, latency, reliability, cost efficiency.

Why Serving Infrastructure Matters

• Production Ready: Models need reliability, not just demos.
• Scale: Handle thousands of concurrent users.
• Cost Control: Optimize GPU utilization and expenses.
• Integration: Clean APIs for application developers.
• Monitoring: Track performance, usage, and errors.

Serving Architecture

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">LLM Deployment — From Weights to Production API</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">model artifacts → optimization → containerize → serve → scale → monitor — the MLOps pipeline for LLMs</text>

  <!-- === TOP: Deployment pipeline === -->
  <rect x="25" y="62" width="710" height="95" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="380" y="80" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">Deployment Pipeline</text>

  <!-- Stages -->
  <rect x="40" y="92" width="80" height="32" rx="3" fill="#1a1520" stroke="#a78bfa" stroke-width="0.8"/>
  <text x="80" y="111" fill="#c4b5fd" font-size="8" text-anchor="middle">Model weights</text>
  <text x="80" y="123" fill="#6b7684" font-size="6.5" text-anchor="middle">HF / S3</text>

  <path d="M123,108 L137,108" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="135,105 141,108 135,111" fill="#8b98a5"/>

  <rect x="144" y="92" width="80" height="32" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="0.8"/>
  <text x="184" y="107" fill="#fbbf24" font-size="8" text-anchor="middle">Quantize</text>
  <text x="184" y="119" fill="#6b7684" font-size="6.5" text-anchor="middle">AWQ/GPTQ/FP8</text>

  <path d="M227,108 L241,108" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="239,105 245,108 239,111" fill="#8b98a5"/>

  <rect x="248" y="92" width="80" height="32" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.8"/>
  <text x="288" y="107" fill="#93c5fd" font-size="8" text-anchor="middle">Compile</text>
  <text x="288" y="119" fill="#6b7684" font-size="6.5" text-anchor="middle">TRT-LLM / vLLM</text>

  <path d="M331,108 L345,108" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="343,105 349,108 343,111" fill="#8b98a5"/>

  <rect x="352" y="92" width="80" height="32" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <text x="392" y="107" fill="#6ee7b7" font-size="8" text-anchor="middle">Container</text>
  <text x="392" y="119" fill="#6b7684" font-size="6.5" text-anchor="middle">Docker + GPU</text>

  <path d="M435,108 L449,108" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="447,105 453,108 447,111" fill="#8b98a5"/>

  <rect x="456" y="92" width="80" height="32" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.8"/>
  <text x="496" y="107" fill="#93c5fd" font-size="8" text-anchor="middle">Serve</text>
  <text x="496" y="119" fill="#6b7684" font-size="6.5" text-anchor="middle">REST / gRPC</text>

  <path d="M539,108 L553,108" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="551,105 557,108 551,111" fill="#8b98a5"/>

  <rect x="560" y="92" width="80" height="32" rx="3" fill="#14261f" stroke="#34d399" stroke-width="0.8"/>
  <text x="600" y="107" fill="#6ee7b7" font-size="8" text-anchor="middle">Scale</text>
  <text x="600" y="119" fill="#6b7684" font-size="6.5" text-anchor="middle">K8s + autoscale</text>

  <path d="M643,108 L657,108" fill="none" stroke="#8b98a5" stroke-width="0.7"/>
  <polygon points="655,105 661,108 655,111" fill="#8b98a5"/>

  <rect x="664" y="92" width="56" height="32" rx="3" fill="#1a0f0f" stroke="#f87171" stroke-width="0.8"/>
  <text x="692" y="107" fill="#f87171" font-size="8" text-anchor="middle">Monitor</text>
  <text x="692" y="119" fill="#6b7684" font-size="6.5" text-anchor="middle">P99/cost</text>

  <text x="380" y="148" fill="#6b7684" font-size="8" text-anchor="middle">Typical timeline: model ready → production API in 1-3 days with vLLM; 1-2 weeks with custom TRT-LLM optimization</text>

  <!-- === MIDDLE LEFT: Infrastructure options === -->
  <rect x="25" y="165" width="350" height="135" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="183" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Infrastructure Options</text>

  <text x="45" y="203" fill="#60a5fa" font-size="8.5" font-weight="600">Managed API</text>
  <text x="135" y="203" fill="#8b98a5" font-size="8.5">OpenAI, Anthropic, Google (easiest)</text>
  <text x="45" y="221" fill="#34d399" font-size="8.5" font-weight="600">Serverless GPU</text>
  <text x="150" y="221" fill="#8b98a5" font-size="8.5">Replicate, Modal, Together (fast start)</text>
  <text x="45" y="239" fill="#fbbf24" font-size="8.5" font-weight="600">Dedicated GPU</text>
  <text x="150" y="239" fill="#8b98a5" font-size="8.5">RunPod, Lambda, CoreWeave (control)</text>
  <text x="45" y="257" fill="#c4b5fd" font-size="8.5" font-weight="600">Cloud VMs</text>
  <text x="120" y="257" fill="#8b98a5" font-size="8.5">AWS p5, GCP a3, Azure ND (scale)</text>
  <text x="45" y="275" fill="#f87171" font-size="8.5" font-weight="600">On-prem</text>
  <text x="105" y="275" fill="#8b98a5" font-size="8.5">own H100 cluster (data sovereignty)</text>
  <text x="45" y="293" fill="#8b98a5" font-size="8.5" font-weight="600">Edge</text>
  <text x="80" y="293" fill="#8b98a5" font-size="8.5">llama.cpp, MLC-LLM, Ollama (local)</text>

  <!-- === MIDDLE RIGHT: Key metrics === -->
  <rect x="390" y="165" width="345" height="135" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="562" y="183" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Production SLA Metrics</text>

  <text x="410" y="205" fill="#60a5fa" font-size="9" font-weight="600">TTFT</text>
  <text x="450" y="205" fill="#8b98a5" font-size="8.5">time to first token (&lt; 500ms target)</text>
  <text x="410" y="223" fill="#34d399" font-size="9" font-weight="600">TPS</text>
  <text x="445" y="223" fill="#8b98a5" font-size="8.5">tokens per second per user (30-80)</text>
  <text x="410" y="241" fill="#fbbf24" font-size="9" font-weight="600">Throughput</text>
  <text x="480" y="241" fill="#8b98a5" font-size="8.5">tokens/s/GPU (2000-5000 batch)</text>
  <text x="410" y="259" fill="#f87171" font-size="9" font-weight="600">P99 latency</text>
  <text x="490" y="259" fill="#8b98a5" font-size="8.5">tail latency under load</text>
  <text x="410" y="277" fill="#c4b5fd" font-size="9" font-weight="600">Cost</text>
  <text x="445" y="277" fill="#8b98a5" font-size="8.5">$/1M output tokens (target &lt; $1 for OSS)</text>
  <text x="410" y="295" fill="#8b98a5" font-size="9" font-weight="600">Uptime</text>
  <text x="460" y="295" fill="#8b98a5" font-size="8.5">99.9% SLA, graceful degradation</text>

  <!-- === BOTTOM: Optimization stack === -->
  <rect x="25" y="310" width="710" height="90" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="328" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Production Optimization Checklist</text>

  <text x="45" y="350" fill="#60a5fa" font-size="8.5">□ Quantize (FP8/INT4) — 2-4× memory savings</text>
  <text x="45" y="366" fill="#34d399" font-size="8.5">□ Continuous batching — 5-10× throughput vs naive</text>
  <text x="45" y="382" fill="#fbbf24" font-size="8.5">□ Prefix caching — save 30-50% for shared prompts</text>

  <text x="390" y="350" fill="#c4b5fd" font-size="8.5">□ Tensor parallel — split model across GPUs</text>
  <text x="390" y="366" fill="#f87171" font-size="8.5">□ CUDA graphs — eliminate kernel launch overhead</text>
  <text x="390" y="382" fill="#8b98a5" font-size="8.5">□ Speculative decoding — 2-3× decode speedup</text>

  <!-- Key insight -->
  <rect x="25" y="408" width="710" height="26" rx="4" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="425" fill="#fbbf24" font-size="9.5" text-anchor="middle">The fastest path to production: vLLM + Docker + Kubernetes + quantized weights = reliable LLM API in hours.</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">LLM deployment is GPU memory management: fit the model, maximize batch size, minimize time-per-token.</text>
</svg>

Serving Frameworks

Framework     | Strengths                    | Best For
--------------|------------------------------|--------------------
vLLM          | PagedAttention, fastest OSS  | High-volume serving
TGI           | HuggingFace, production      | HF ecosystem
TensorRT-LLM  | NVIDIA optimized, fastest    | NVIDIA hardware
Triton        | Multi-model, enterprise      | Complex pipelines
llama.cpp     | CPU/edge, portable           | Local deployment
Ollama        | Simple local, CLI            | Developer setup

API Design Patterns

Chat Completions API (OpenAI-compatible):

POST /v1/chat/completions
{
    "model": "llama-3.1-70b",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing"}
    ],
    "temperature": 0.7,
    "max_tokens": 1000,
    "stream": true
}

Streaming Response (SSE):

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":"Quantum"}}]}
data: {"id":"chatcmpl-123","choices":[{"delta":{"content":" computing"}}]}
data: {"id":"chatcmpl-123","choices":[{"delta":{"content":" is"}}]}
...
data: [DONE]

Key API Features

• Streaming: SSE/WebSocket for token-by-token delivery.
• Function Calling: Structured tool use capabilities.
• JSON Mode: Guaranteed valid JSON output.
• Logprobs: Token probabilities for confidence.
• Stop Sequences: Custom stopping conditions.
• Seed: Reproducible generation.

Production Considerations

Rate Limiting:

Strategies:
- Requests per minute (RPM)
- Tokens per minute (TPM)
- Per-user quotas
- Per-tier limits

Cost Management:

• Track tokens/cost per user/team.
• Set spend limits and alerts.
• Optimize batch vs. real-time.
• Cache common queries.

Reliability:

• Health checks and auto-restart.
• Graceful degradation.
• Multi-region deployment.
• Automatic failover.

Deployment Options

Managed APIs (Zero infrastructure):

• OpenAI, Anthropic, Google APIs.
• Highest simplicity, lowest control.

Serverless GPU (Minimal ops):

• Replicate, Modal, RunPod, Together.
• Pay per use, automatic scaling.

Self-Hosted Cloud (Full control):

• AWS/GCP/Azure GPU instances.
• Kubernetes with GPU operators.
• Higher ops burden, more control.

On-Premise (Maximum control):

• NVIDIA DGX systems.
• Air-gapped environments.
• Full data sovereignty.

LLM serving and APIs is where AI capabilities meet product requirements — robust serving infrastructure determines whether AI features are reliable and cost-effective or fragile and expensive, making serving engineering essential for any production AI application.