p50/p95/p99 latency
**p50/p95/p99 latency** refers to **percentile latency metrics** that describe the distribution of response times across all requests. Unlike averages, percentiles reveal how **different subsets of users** experience the system, making them essential for meaningful performance monitoring.
**What The Percentiles Mean**
- **p50 (50th percentile / median)**: 50% of requests complete faster than this value. Represents the **typical user experience**.
- **p95 (95th percentile)**: 95% of requests complete faster. Only 5% of users experience worse latency. This is the **standard SLO target** for most services.
- **p99 (99th percentile)**: 99% of requests complete faster. Only 1% experience worse latency. Captures **tail latency** problems.
- **p99.9**: 99.9% are faster — used for critical systems where even rare slowness is unacceptable.
**Why Percentiles, Not Averages?**
Consider 100 requests: 99 complete in 100ms, 1 takes 10,000ms.
- **Average**: 199ms — looks fine!
- **p99**: 10,000ms — reveals that 1 in 100 users waits 10 seconds.
Averages **hide tail latency** problems that significantly impact user experience. In high-traffic systems, even p99 affects thousands of users daily.
**Typical SLOs for LLM Applications**
- **Streaming TTFT**: p50 < 200ms, p95 < 500ms, p99 < 1,000ms
- **Total Response**: p50 < 2s, p95 < 5s, p99 < 10s
- **API-to-API**: p50 < 1s, p95 < 3s, p99 < 5s
**Causes of High Tail Latency**
- **Garbage Collection Pauses**: JVM or Python GC can cause occasional spikes.
- **Cold Starts**: First request to a new instance is significantly slower.
- **Resource Contention**: GPU memory pressure, CPU scheduling conflicts.
- **Long Outputs**: Requests generating very long responses naturally take longer.
- **Batch Queuing**: Continuous batching can delay individual requests when the batch is full.
**Monitoring**
- Use **histograms** (Prometheus) or **distribution metrics** (Datadog) to compute percentiles.
- Display p50, p95, and p99 on the same graph to visualize the spread.
- Alert when p95 or p99 exceed SLO thresholds for more than 5 minutes.
Percentile latency metrics are the **gold standard** for performance monitoring — any serious production system tracks at least p50, p95, and p99.