Time Per Output Token (TPOT) measures the average time the model takes to generate each successive token after the first token has been produced. While TTFT measures how quickly output begins, TPOT determines the streaming speed — how fast the text appears to flow once generation has started.

What Determines TPOT

• Decode Phase: Each output token requires a forward pass through the entire model, but only for a single token position (unlike prefill which processes all input tokens at once). This makes individual decode steps fast but they add up.
• Memory Bandwidth: Decode is typically memory-bandwidth bound rather than compute-bound — the GPU spends most of its time loading model weights from memory rather than doing arithmetic.
• KV Cache Size: As more tokens are generated, the key-value cache grows, requiring more memory reads during attention computation.
• Batch Size: Serving multiple requests simultaneously improves GPU utilization but can increase per-request TPOT.

Typical TPOT Values

• 7B model on H100: 5–15 ms/token (~65–200 tokens/second)
• 70B model on H100: 20–50 ms/token (~20–50 tokens/second)
• Human reading speed: ~250 words/minute ≈ ~5.5 tokens/second — so even moderate TPOT values produce text faster than humans can read.

Optimization Approaches

• Quantization: Reducing model precision to INT8/INT4 decreases memory read volume, directly improving TPOT.
• Speculative Decoding: A small draft model predicts several tokens at once, and the large model verifies them in a single forward pass.
• Paged Attention (vLLM): Efficient KV cache memory management prevents fragmentation and wasted GPU memory.
• Tensor Parallelism: Splitting the model across multiple GPUs reduces per-GPU memory reads.

For user-facing applications, keeping TPOT below ~30 ms ensures text appears to stream smoothly and naturally.