benchmarking llm
**Benchmarking LLM performance** is the **systematic measurement of inference speed, throughput, and quality** — using standardized tests to measure time-to-first-token (TTFT), tokens-per-second, concurrent capacity, and response quality, enabling informed decisions about model selection, infrastructure sizing, and optimization priorities.
**What Is LLM Benchmarking?**
- **Definition**: Measuring LLM system performance under controlled conditions.
- **Metrics**: Latency, throughput, quality, cost.
- **Purpose**: Compare options, identify bottlenecks, validate optimizations.
- **Types**: Synthetic load tests and real-world workload simulations.
**Why Benchmarking Matters**
- **Model Selection**: Choose between GPT-4o, Claude, Llama based on data.
- **Capacity Planning**: Know how many GPUs needed for target load.
- **Optimization**: Measure impact of changes.
- **SLA Validation**: Ensure system meets latency requirements.
- **Cost Analysis**: Understand cost-per-query at different scales.
**Key Performance Metrics**
**Latency Metrics**:
```
TTFT (Time to First Token):
- Measures prefill latency
- Target: <500ms for interactive
- Critical for perceived responsiveness
TPOT (Time Per Output Token):
- Decode latency per token
- Target: <50ms for smooth streaming
- Lower = faster generation
E2E (End-to-End):
- Total response time
- E2E = TTFT + (TPOT × output_tokens)
```
**Throughput Metrics**:
```
Tokens/Second:
- Total generation throughput
- Maximized for batch workloads
Requests/Second:
- Completed requests per second
- Depends on response length
Concurrent Users:
- Simultaneous active requests
- Limited by memory (KV cache)
```
**Percentile Latencies**:
```
P50: Median latency (typical experience)
P95: 95th percentile (most users)
P99: 99th percentile (worst common case)
Max: Absolute worst case
Target: P99 < 2× P50 for consistent experience
```
**Benchmarking Tools**
```
Tool | Type | Features
------------|----------------|-------------------------
LLMPerf | LLM-specific | TTFT, TPOT, concurrency
k6 | Load testing | Flexible scripting
Locust | Load testing | Python-based, distributed
hey | HTTP benchmark | Simple, quick tests
wrk | HTTP benchmark | High performance
Custom | Any | Precise control
```
**Simple Benchmark Script**:
```python
import time
import statistics
from openai import OpenAI
client = OpenAI()
def benchmark_request(prompt):
start = time.time()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": prompt}],
stream=True
)
first_token_time = None
token_count = 0
for chunk in response:
if first_token_time is None:
first_token_time = time.time()
if chunk.choices[0].delta.content:
token_count += 1
end = time.time()
return {
"ttft": first_token_time - start,
"total_time": end - start,
"tokens": token_count,
"tpot": (end - first_token_time) / token_count
}
# Run multiple iterations
results = [benchmark_request("Explain quantum computing") for _ in range(10)]
# Calculate statistics
ttfts = [r["ttft"] for r in results]
print(f"TTFT P50: {statistics.median(ttfts):.3f}s")
print(f"TTFT P95: {sorted(ttfts)[int(len(ttfts)*0.95)]:.3f}s")
```
**Load Testing with Locust**:
```python
from locust import HttpUser, task, between
class LLMUser(HttpUser):
wait_time = between(1, 3)
@task
def generate_response(self):
self.client.post(
"/v1/chat/completions",
json={
"model": "gpt-4o",
"messages": [{"role": "user", "content": "Hello!"}]
},
headers={"Authorization": "Bearer ..."}
)
```
**Benchmark Methodology**
```svg
```
**Best Practices**
- **Warm Up**: Run requests before measuring to warm caches.
- **Realistic Load**: Use production-like prompt distributions.
- **Sufficient Duration**: Run long enough for stable results.
- **Monitor System**: Watch GPU utilization, memory during test.
- **Multiple Runs**: Account for variance in results.
- **Document Everything**: Record versions, configurations, conditions.
Benchmarking LLM performance is **essential for production planning** — without rigorous measurement, teams make infrastructure decisions based on hope rather than data, leading to either overspending or underprovisioning that impacts user experience.