Latency monitoring is the practice of continuously tracking the time it takes for an AI system to process requests and deliver responses. For LLM applications, latency directly impacts user experience — slow responses feel broken, while fast responses feel like natural conversation.

Key Latency Metrics

• TTFT (Time to First Token): Time from request submission to receiving the first token of the response. Critical for perceived speed in streaming applications.
• TPOT (Time Per Output Token): Average time to generate each subsequent token. Determines the speed of streaming text appearance.
• Total Latency: End-to-end time from request to complete response. Important for non-streaming and API-to-API calls.
• Queue Wait Time: Time spent waiting in the request queue before inference begins.
• Preprocessing Latency: Time for input validation, tokenization, and prompt construction.
• Retrieval Latency: Time for RAG vector search and document retrieval.

Percentile Metrics

• p50 (Median): The typical user experience — 50% of requests are faster than this.
• p95: 95% of requests are faster — captures most users' experience.
• p99: 99% of requests are faster — captures the worst common experience.
• p99.9: Extreme tail latency — important for SLA compliance.

Why Percentiles Matter More Than Averages

Averages mask problems — a system with 100ms average latency might have p99 of 5,000ms. One in 100 users experiences a 50× slower response. Averages look fine; percentiles reveal the truth.

Monitoring Best Practices

• Set SLOs: Define Service Level Objectives (e.g., "p95 TTFT < 500ms, p99 total latency < 10s").
• Alert on SLO Breaches: Trigger alerts when latency SLOs are violated for a sustained period.
• Break Down by Component: Monitor latency at each pipeline stage to identify bottlenecks.
• Segment by Request Type: Simple queries vs. complex reasoning, short vs. long responses.
• Dashboard Visualization: Time-series graphs of p50/p95/p99 with deployment annotations.

Common Latency Issues in LLM Systems

• Cold Start: First request after scaling up is slow due to model loading.
• Long Contexts: Latency scales with context length (quadratically for attention).
• Batch Contention: Large batch sizes improve throughput but increase individual request latency.

Latency monitoring is the most user-visible metric for AI applications — users forgive occasional errors but not consistent slowness.