Benchmarking LLM performance is the systematic measurement of inference speed, throughput, and quality — using standardized tests to measure time-to-first-token (TTFT), tokens-per-second, concurrent capacity, and response quality, enabling informed decisions about model selection, infrastructure sizing, and optimization priorities.

What Is LLM Benchmarking?

• Definition: Measuring LLM system performance under controlled conditions.
• Metrics: Latency, throughput, quality, cost.
• Purpose: Compare options, identify bottlenecks, validate optimizations.
• Types: Synthetic load tests and real-world workload simulations.

Why Benchmarking Matters

• Model Selection: Choose between GPT-4o, Claude, Llama based on data.
• Capacity Planning: Know how many GPUs needed for target load.
• Optimization: Measure impact of changes.
• SLA Validation: Ensure system meets latency requirements.
• Cost Analysis: Understand cost-per-query at different scales.

Key Performance Metrics

Latency Metrics:

TTFT (Time to First Token):
- Measures prefill latency
- Target: <500ms for interactive
- Critical for perceived responsiveness

TPOT (Time Per Output Token):
- Decode latency per token
- Target: <50ms for smooth streaming
- Lower = faster generation

E2E (End-to-End):
- Total response time
- E2E = TTFT + (TPOT × output_tokens)

Throughput Metrics:

Tokens/Second:
- Total generation throughput
- Maximized for batch workloads

Requests/Second:
- Completed requests per second
- Depends on response length

Concurrent Users:
- Simultaneous active requests
- Limited by memory (KV cache)

Percentile Latencies:

P50: Median latency (typical experience)
P95: 95th percentile (most users)
P99: 99th percentile (worst common case)
Max: Absolute worst case

Target: P99 < 2× P50 for consistent experience

Benchmarking Tools

Tool        | Type           | Features
------------|----------------|-------------------------
LLMPerf     | LLM-specific   | TTFT, TPOT, concurrency
k6          | Load testing   | Flexible scripting
Locust      | Load testing   | Python-based, distributed
hey         | HTTP benchmark | Simple, quick tests
wrk         | HTTP benchmark | High performance
Custom      | Any            | Precise control

Simple Benchmark Script:

import time
import statistics
from openai import OpenAI

client = OpenAI()

def benchmark_request(prompt):
    start = time.time()
    
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        stream=True
    )
    
    first_token_time = None
    token_count = 0
    
    for chunk in response:
        if first_token_time is None:
            first_token_time = time.time()
        if chunk.choices[0].delta.content:
            token_count += 1
    
    end = time.time()
    
    return {
        "ttft": first_token_time - start,
        "total_time": end - start,
        "tokens": token_count,
        "tpot": (end - first_token_time) / token_count
    }

# Run multiple iterations
results = [benchmark_request("Explain quantum computing") for _ in range(10)]

# Calculate statistics
ttfts = [r["ttft"] for r in results]
print(f"TTFT P50: {statistics.median(ttfts):.3f}s")
print(f"TTFT P95: {sorted(ttfts)[int(len(ttfts)*0.95)]:.3f}s")

Load Testing with Locust:

from locust import HttpUser, task, between

class LLMUser(HttpUser):
    wait_time = between(1, 3)
    
    @task
    def generate_response(self):
        self.client.post(
            "/v1/chat/completions",
            json={
                "model": "gpt-4o",
                "messages": [{"role": "user", "content": "Hello!"}]
            },
            headers={"Authorization": "Bearer ..."}
        )

Benchmark Methodology

┌─────────────────────────────────────────────────────┐
│ 1. Define Test Scenarios                            │
│    - Realistic prompts (varied lengths)             │
│    - Expected output lengths                        │
│    - Concurrency patterns                           │
├─────────────────────────────────────────────────────┤
│ 2. Establish Baseline                               │
│    - Warm up system                                 │
│    - Run baseline at low load                       │
│    - Record all metrics                             │
├─────────────────────────────────────────────────────┤
│ 3. Stress Test                                      │
│    - Gradually increase load                        │
│    - Find breaking point                            │
│    - Identify bottleneck                            │
├─────────────────────────────────────────────────────┤
│ 4. Analyze Results                                  │
│    - Plot latency vs. load                          │
│    - Calculate cost per request                     │
│    - Compare to requirements                        │
└─────────────────────────────────────────────────────┘

Best Practices

• Warm Up: Run requests before measuring to warm caches.
• Realistic Load: Use production-like prompt distributions.
• Sufficient Duration: Run long enough for stable results.
• Monitor System: Watch GPU utilization, memory during test.
• Multiple Runs: Account for variance in results.
• Document Everything: Record versions, configurations, conditions.

Benchmarking LLM performance is essential for production planning — without rigorous measurement, teams make infrastructure decisions based on hope rather than data, leading to either overspending or underprovisioning that impacts user experience.