latency

**Latency** in the context of AI and LLM deployment refers to the **time delay** between sending a request to a model and receiving the beginning of its response. It is one of the most critical performance metrics for any real-time AI application. **Components of LLM Latency** - **Network Latency**: The round-trip time for the request to reach the inference server and the response to return. Typically **1–50 ms** depending on geography and infrastructure. - **Queue Wait Time**: Time spent waiting for a GPU to become available if the system is under load. - **Prefill Latency**: Time to process all **input tokens** (the prompt) through the model. Scales with prompt length. - **Time to First Token (TTFT)**: The total delay before the first output token is generated — includes network + queue + prefill time. - **Decode Latency**: Time to generate each subsequent output token. Determines the perceived **streaming speed**. **Typical Latency Targets** - **Interactive Chat**: TTFT under **500 ms**, decode at **30+ tokens/second** for a smooth conversational experience. - **API Calls**: End-to-end response within **1–5 seconds** for most applications. - **Real-Time Systems**: Sub-**100 ms** TTFT required for voice assistants, gaming, and robotics. **Optimization Techniques** - **KV Cache**: Stores previously computed key-value pairs to avoid redundant computation during autoregressive decoding. - **Speculative Decoding**: Uses a smaller draft model to predict multiple tokens in parallel, verified by the main model. - **Model Distillation**: Smaller, faster models trained to mimic larger ones. - **Hardware Upgrades**: Faster GPUs with higher memory bandwidth (like **NVIDIA H100/H200**) directly reduce latency.