latency
**Latency optimization** is the **systematic reduction of response time in LLM inference** — minimizing the delay between user input and AI response through techniques like quantization, KV cache optimization, speculative decoding, and model architecture choices, critical for real-time interactive applications.
**What Is Latency in LLM Inference?**
- **Definition**: Time from request submission to complete response.
- **Components**: Queue time + prefill (TTFT) + decode (TPOT × tokens).
- **Target**: Interactive applications need <100ms TTFT, <50ms TPOT.
- **Challenge**: Balance latency with throughput and cost.
**Why Latency Matters**
- **User Experience**: Slow responses frustrate users (<200ms feels instant).
- **Conversational Flow**: Real-time chat requires low latency.
- **Competitive Advantage**: Faster AI feels smarter and more capable.
- **Use Cases**: Autocomplete, coding assistants, voice need sub-second.
- **Throughput Trade-off**: Lower latency often means lower throughput.
**Latency Breakdown**
**Key Metrics**:
```svg
```
**Latency Targets by Use Case**:
```
Use Case | TTFT Target | TPOT Target
-------------------|-------------|-------------
Voice assistant | <300ms | <40ms
Chat interface | <500ms | <50ms
Code completion | <200ms | <30ms
Batch processing | N/A | Maximize throughput
```
**Optimization Techniques**
**Quantization**:
- INT8/INT4 weights reduce memory bandwidth requirements.
- 2-4× speedup with minimal quality loss.
- AWQ, GPTQ, bitsandbytes implementations.
- FP8 on modern GPUs (H100) for best speed/quality.
**KV Cache Optimizations**:
- **PagedAttention**: Reduce memory fragmentation.
- **Quantized KV**: INT8/INT4 cache values.
- **Prefix Caching**: Reuse KV for common system prompts.
- **Sliding Window**: Limit attention span (Mistral).
**Speculative Decoding**:
```
1. Small "draft" model generates N candidate tokens quickly
2. Large "target" model verifies all N in parallel
3. Accept matching tokens, reject at first mismatch
4. Net speed: ~2-3× faster for matching drafts
Example: 7B draft + 70B verify = faster than 70B alone
```
**Model Architecture**:
- **GQA/MQA**: Fewer Key-Value heads = faster decode.
- **Smaller Models**: Latency scales with model size.
- **MoE**: Only activate subset of parameters.
- **Early Exit**: Stop at confident predictions.
**Attention Optimizations**:
- **Flash Attention**: Fused kernel, IO-aware.
- **Flash Attention 2/3**: Further optimized versions.
- **Paged Attention**: Memory-efficient for variable lengths.
**Infrastructure Optimizations**
**Hardware Selection**:
```
GPU | Memory BW | Typical TPOT (7B)
----------------|------------|------------------
RTX 4090 | 1 TB/s | 15-25ms
A100 (80GB) | 2 TB/s | 10-15ms
H100 (80GB) | 3.35 TB/s | 6-10ms
H200 (141GB) | 4.8 TB/s | 4-7ms
```
**Network & Infrastructure**:
- Deploy close to users (edge, CDN).
- Use gRPC over REST for lower overhead.
- Connection pooling, keep-alive.
- Streaming responses (SSE) for perceived speed.
**Measurement & Monitoring**
- **P50/P95/P99 Latencies**: Distribution matters, not just average.
- **Real-time Dashboards**: Monitor TTFT, TPOT, queue depth.
- **Load Testing**: Stress test before production.
- **Alerting**: Detect latency regressions quickly.
Latency optimization is **essential for user-facing AI applications** — the difference between a 500ms and 2000ms response time determines whether AI feels like a helpful assistant or a frustrating bottleneck, making latency engineering critical for any interactive AI product.