time to first token (ttft)

**Time to First Token (TTFT)** measures the **latency** from the moment a user sends a request to when the **first token** of the model's response begins streaming back. It is the single most important responsiveness metric for interactive LLM applications because it determines how long users wait before seeing any output. **What Contributes to TTFT** - **Prefill Phase**: The model must process the entire input prompt through all transformer layers before generating the first output token. This is the dominant component — longer prompts mean higher TTFT. - **Network Round-Trip**: Time for the request to travel to the server and the first token to return. - **Queue Delay**: If GPUs are busy handling other requests, the new request waits in a queue. - **Model Loading**: For serverless deployments, there may be a **cold start** penalty if the model needs to be loaded into GPU memory. **Typical TTFT Values** - **Small Models (7B)**: **50–200 ms** on modern hardware with short prompts. - **Large Models (70B+)**: **200 ms–2 seconds** depending on prompt length and hardware. - **Very Long Contexts (100K+ tokens)**: Can reach **5–30+ seconds** due to the quadratic attention computation during prefill. **Optimization Strategies** - **Chunked Prefill**: Break the prefill computation into chunks, interleaving with decode requests to keep the system responsive. - **Prompt Caching**: Cache prefill results for common system prompts so they don't need reprocessing. - **Faster Hardware**: GPUs with higher **FLOPS** (like H100 vs A100) directly reduce prefill time. - **Prefix Sharing**: When multiple requests share the same prompt prefix, compute it once and reuse. TTFT is especially critical for **user-facing chat applications** where perceived responsiveness directly impacts user satisfaction.