Speculative Decoding is the LLM inference acceleration technique that uses a small, fast "draft" model to generate multiple candidate tokens in parallel, which the large "target" model then verifies in a single forward pass — achieving 2-3x speedup with mathematically guaranteed identical output distribution to standard autoregressive generation from the target model alone.

Why Standard LLM Inference Is Slow

Autoregressive generation is inherently sequential: each token depends on all previous tokens, so the model performs one forward pass per token. For large models (70B+ parameters), each forward pass takes 50-200ms, and most of that time is spent loading model weights from memory (memory-bandwidth-bound). The GPU's compute units are severely underutilized — generating one token at a time wastes the massive parallelism GPUs provide.

How Speculative Decoding Works

1. Draft: A small model (e.g., 1-7B parameters) generates K candidate tokens autoregressively (fast, since the model is small). These K tokens represent a speculative continuation. 2. Verify: The large target model processes the entire draft sequence in a single forward pass (just like processing a prompt — fully parallel). It computes the probability distribution at each position. 3. Accept/Reject: Starting from the first draft token, each is accepted if the target model's probability for that token is sufficiently high relative to the draft model's probability. A modified rejection sampling scheme ensures the accepted tokens follow exactly the target model's distribution. The first rejected token is resampled from an adjusted distribution. 4. Repeat: The process continues from the last accepted token.

Why It Produces Identical Outputs

The acceptance criterion uses a specific probability ratio: accept token x with probability min(1, p_target(x) / p_draft(x)). If rejected, sample from the residual distribution (p_target - p_draft), normalized. This is mathematically proven to reproduce the exact target distribution — there is zero quality degradation.

Speedup Analysis

If the draft model agrees with the target model on ~70% of tokens (common for well-chosen draft/target pairs), and draft length K=5, the expected accepted tokens per verification is ~3.5. Since verification costs roughly the same as generating one token (both are one forward pass), the effective speedup is ~3.5x.

Variants

• Self-Speculative Decoding: Uses early exit from the target model itself (e.g., output from layer 8 of a 32-layer model) as the draft, eliminating the need for a separate draft model.
• Medusa: Adds multiple parallel prediction heads to the target model, each predicting a different future token position. No separate draft model needed.
• EAGLE: Uses a lightweight autoregressive head on top of the target model's hidden states for more accurate drafting.
• Lookahead Decoding: Generates multiple n-gram candidates in parallel using Jacobi iteration, verifying them in a single forward pass.

Speculative Decoding is the free lunch of LLM inference — achieving substantial speedup with zero quality loss by exploiting the asymmetry between sequential generation cost and parallel verification cost.