Speculative Decoding is the inference acceleration technique that uses a smaller, faster draft model to propose multiple tokens in parallel, which the larger target model then verifies in a single forward pass — exploiting the fact that verification of N tokens (one forward pass through the target) is much cheaper than generating N tokens autoregressively (N forward passes), achieving 2-3× speedup with mathematically guaranteed identical output distribution to the original model, making it one of the few "free lunch" optimizations for LLM inference.

The Autoregressive Bottleneck

Standard autoregression (100 tokens):
  Token 1 → [Full model forward pass] → Token 2 → [Full model forward pass] → ...
  100 sequential forward passes, each memory-bandwidth-bound
  Time: 100 × latency_per_token

Speculative decoding (100 tokens):
  Draft model proposes K tokens in parallel
  Target model verifies K tokens in one forward pass
  Accept all correct tokens, regenerate from first wrong one
  Time: ~(100/K) × latency_per_token (if acceptance rate is high)

How It Works

1. Draft model generates K candidate tokens:
   [The] → draft → [quick] [brown] [fox] [jumped] [over]

2. Target model scores ALL candidates in one forward pass:
   P_target(quick|The) = 0.85        (draft said 0.80) → Accept
   P_target(brown|The quick) = 0.90   (draft said 0.88) → Accept  
   P_target(fox|...brown) = 0.75      (draft said 0.70) → Accept
   P_target(jumped|...fox) = 0.30     (draft said 0.60) → Reject!

3. Accept first 3 tokens, resample token 4 from adjusted distribution
   Output: [The] [quick] [brown] [fox] [leaped]
   Net gain: 3 tokens verified in 1 target pass instead of 3 passes

Mathematical Guarantee

• Acceptance criterion uses modified rejection sampling.
• If P_draft(x) ≤ P_target(x): Always accept.
• If P_draft(x) > P_target(x): Accept with probability P_target(x)/P_draft(x).
• On rejection: Sample from residual distribution (P_target - P_draft).
• Theorem: Output distribution is exactly P_target regardless of draft model quality.

Draft Model Strategies

Performance Results

When Speculative Decoding Helps Most

• Batch size 1 (interactive): Maximum benefit (memory-bandwidth bound).
• Code generation: High acceptance rate (code is predictable).
• Translation: Draft model easily approximates structure.
• Large batch: Less benefit (compute-bound, not bandwidth-bound).

Speculative decoding is the most important inference optimization for interactive LLM serving — by turning the sequential token-generation bottleneck into a parallel verify-and-accept loop, speculative decoding delivers 2-3× latency reduction with zero quality degradation, making it essential infrastructure for real-time AI applications from chatbots to code assistants, where every millisecond of response time directly impacts user experience.