Latency in the context of AI and LLM deployment refers to the time delay between sending a request to a model and receiving the beginning of its response. It is one of the most critical performance metrics for any real-time AI application.
Components of LLM Latency
- Network Latency: The round-trip time for the request to reach the inference server and the response to return. Typically 1–50 ms depending on geography and infrastructure.
- Queue Wait Time: Time spent waiting for a GPU to become available if the system is under load.
- Prefill Latency: Time to process all input tokens (the prompt) through the model. Scales with prompt length.
- Time to First Token (TTFT): The total delay before the first output token is generated — includes network + queue + prefill time.
- Decode Latency: Time to generate each subsequent output token. Determines the perceived streaming speed.
Typical Latency Targets
- Interactive Chat: TTFT under 500 ms, decode at 30+ tokens/second for a smooth conversational experience.
- API Calls: End-to-end response within 1–5 seconds for most applications.
- Real-Time Systems: Sub-100 ms TTFT required for voice assistants, gaming, and robotics.
Optimization Techniques
- KV Cache: Stores previously computed key-value pairs to avoid redundant computation during autoregressive decoding.
- Speculative Decoding: Uses a smaller draft model to predict multiple tokens in parallel, verified by the main model.
- Model Distillation: Smaller, faster models trained to mimic larger ones.
- Hardware Upgrades: Faster GPUs with higher memory bandwidth (like NVIDIA H100/H200) directly reduce latency.