Time to First Token (TTFT) measures the latency from the moment a user sends a request to when the first token of the model's response begins streaming back. It is the single most important responsiveness metric for interactive LLM applications because it determines how long users wait before seeing any output.

What Contributes to TTFT

• Prefill Phase: The model must process the entire input prompt through all transformer layers before generating the first output token. This is the dominant component — longer prompts mean higher TTFT.
• Network Round-Trip: Time for the request to travel to the server and the first token to return.
• Queue Delay: If GPUs are busy handling other requests, the new request waits in a queue.
• Model Loading: For serverless deployments, there may be a cold start penalty if the model needs to be loaded into GPU memory.

Typical TTFT Values

• Small Models (7B): 50–200 ms on modern hardware with short prompts.
• Large Models (70B+): 200 ms–2 seconds depending on prompt length and hardware.
• Very Long Contexts (100K+ tokens): Can reach 5–30+ seconds due to the quadratic attention computation during prefill.

Optimization Strategies

• Chunked Prefill: Break the prefill computation into chunks, interleaving with decode requests to keep the system responsive.
• Prompt Caching: Cache prefill results for common system prompts so they don't need reprocessing.
• Faster Hardware: GPUs with higher FLOPS (like H100 vs A100) directly reduce prefill time.
• Prefix Sharing: When multiple requests share the same prompt prefix, compute it once and reuse.

TTFT is especially critical for user-facing chat applications where perceived responsiveness directly impacts user satisfaction.