time per output token (tpot)
**Time Per Output Token (TPOT)** measures the **average time** the model takes to generate each successive token after the first token has been produced. While **TTFT** measures how quickly output begins, TPOT determines the **streaming speed** — how fast the text appears to flow once generation has started.
**What Determines TPOT**
- **Decode Phase**: Each output token requires a **forward pass** through the entire model, but only for a single token position (unlike prefill which processes all input tokens at once). This makes individual decode steps fast but they add up.
- **Memory Bandwidth**: Decode is typically **memory-bandwidth bound** rather than compute-bound — the GPU spends most of its time loading model weights from memory rather than doing arithmetic.
- **KV Cache Size**: As more tokens are generated, the **key-value cache** grows, requiring more memory reads during attention computation.
- **Batch Size**: Serving multiple requests simultaneously improves GPU utilization but can increase per-request TPOT.
**Typical TPOT Values**
- **7B model on H100**: **5–15 ms/token** (~65–200 tokens/second)
- **70B model on H100**: **20–50 ms/token** (~20–50 tokens/second)
- **Human reading speed**: ~250 words/minute ≈ ~5.5 tokens/second — so even moderate TPOT values produce text faster than humans can read.
**Optimization Approaches**
- **Quantization**: Reducing model precision to **INT8/INT4** decreases memory read volume, directly improving TPOT.
- **Speculative Decoding**: A small draft model predicts several tokens at once, and the large model verifies them in a single forward pass.
- **Paged Attention (vLLM)**: Efficient KV cache memory management prevents fragmentation and wasted GPU memory.
- **Tensor Parallelism**: Splitting the model across multiple GPUs reduces per-GPU memory reads.
For user-facing applications, keeping TPOT below **~30 ms** ensures text appears to stream smoothly and naturally.