p50/p95/p99 latency refers to percentile latency metrics that describe the distribution of response times across all requests. Unlike averages, percentiles reveal how different subsets of users experience the system, making them essential for meaningful performance monitoring.

What The Percentiles Mean

• p50 (50th percentile / median): 50% of requests complete faster than this value. Represents the typical user experience.
• p95 (95th percentile): 95% of requests complete faster. Only 5% of users experience worse latency. This is the standard SLO target for most services.
• p99 (99th percentile): 99% of requests complete faster. Only 1% experience worse latency. Captures tail latency problems.
• p99.9: 99.9% are faster — used for critical systems where even rare slowness is unacceptable.

Why Percentiles, Not Averages?

Consider 100 requests: 99 complete in 100ms, 1 takes 10,000ms.

• Average: 199ms — looks fine!
• p99: 10,000ms — reveals that 1 in 100 users waits 10 seconds.

Averages hide tail latency problems that significantly impact user experience. In high-traffic systems, even p99 affects thousands of users daily.

Typical SLOs for LLM Applications

• Streaming TTFT: p50 < 200ms, p95 < 500ms, p99 < 1,000ms
• Total Response: p50 < 2s, p95 < 5s, p99 < 10s
• API-to-API: p50 < 1s, p95 < 3s, p99 < 5s

Causes of High Tail Latency

• Garbage Collection Pauses: JVM or Python GC can cause occasional spikes.
• Cold Starts: First request to a new instance is significantly slower.
• Resource Contention: GPU memory pressure, CPU scheduling conflicts.
• Long Outputs: Requests generating very long responses naturally take longer.
• Batch Queuing: Continuous batching can delay individual requests when the batch is full.

Monitoring

• Use histograms (Prometheus) or distribution metrics (Datadog) to compute percentiles.
• Display p50, p95, and p99 on the same graph to visualize the spread.
• Alert when p95 or p99 exceed SLO thresholds for more than 5 minutes.

Percentile latency metrics are the gold standard for performance monitoring — any serious production system tracks at least p50, p95, and p99.