latency monitoring

**Latency monitoring** is the practice of continuously tracking the **time it takes** for an AI system to process requests and deliver responses. For LLM applications, latency directly impacts user experience — slow responses feel broken, while fast responses feel like natural conversation. **Key Latency Metrics** - **TTFT (Time to First Token)**: Time from request submission to receiving the first token of the response. Critical for **perceived speed** in streaming applications. - **TPOT (Time Per Output Token)**: Average time to generate each subsequent token. Determines the speed of streaming text appearance. - **Total Latency**: End-to-end time from request to complete response. Important for non-streaming and API-to-API calls. - **Queue Wait Time**: Time spent waiting in the request queue before inference begins. - **Preprocessing Latency**: Time for input validation, tokenization, and prompt construction. - **Retrieval Latency**: Time for RAG vector search and document retrieval. **Percentile Metrics** - **p50 (Median)**: The typical user experience — 50% of requests are faster than this. - **p95**: 95% of requests are faster — captures most users' experience. - **p99**: 99% of requests are faster — captures the worst common experience. - **p99.9**: Extreme tail latency — important for SLA compliance. **Why Percentiles Matter More Than Averages** Averages mask problems — a system with 100ms average latency might have p99 of 5,000ms. One in 100 users experiences a **50× slower** response. Averages look fine; percentiles reveal the truth. **Monitoring Best Practices** - **Set SLOs**: Define Service Level Objectives (e.g., "p95 TTFT < 500ms, p99 total latency < 10s"). - **Alert on SLO Breaches**: Trigger alerts when latency SLOs are violated for a sustained period. - **Break Down by Component**: Monitor latency at each pipeline stage to identify bottlenecks. - **Segment by Request Type**: Simple queries vs. complex reasoning, short vs. long responses. - **Dashboard Visualization**: Time-series graphs of p50/p95/p99 with deployment annotations. **Common Latency Issues in LLM Systems** - **Cold Start**: First request after scaling up is slow due to model loading. - **Long Contexts**: Latency scales with context length (quadratically for attention). - **Batch Contention**: Large batch sizes improve throughput but increase individual request latency. Latency monitoring is **the most user-visible** metric for AI applications — users forgive occasional errors but not consistent slowness.