speculative decoding

**Speculative Decoding** is the **inference acceleration technique that uses a small, fast draft model to generate multiple candidate tokens in parallel, which are then verified by the large target model in a single forward pass — achieving 2-3x speedup in autoregressive LLM inference without any change to the output distribution, because verification of K draft tokens costs approximately the same as generating one token from the large model**. **The Autoregressive Bottleneck** Standard LLM inference generates one token at a time: each token requires a full forward pass through the model, and the next token depends on the previous one (sequential dependency). For a 70B parameter model, each forward pass takes ~30-50 ms on a single GPU, limiting throughput to ~20-30 tokens/second regardless of available compute — the process is memory-bandwidth bound, not compute bound. **How Speculative Decoding Works** 1. **Draft Phase**: A small model (e.g., 1B parameters, 10x faster) generates K candidate tokens autoregressively: t₁, t₂, ..., tₖ. 2. **Verification Phase**: The large target model processes the original context plus all K draft tokens in a single forward pass (parallel evaluation, like processing a prompt). This produces the target model's probability distributions for each position. 3. **Acceptance/Rejection**: Starting from t₁, each draft token is accepted with probability min(1, p_target(tᵢ)/p_draft(tᵢ)). If a token is rejected, it is resampled from an adjusted distribution. All tokens after a rejection are discarded. 4. **Guarantee**: The acceptance-rejection scheme ensures the output distribution is mathematically identical to sampling directly from the target model — zero quality degradation. **Why It Works** LLM inference is memory-bandwidth bound: loading the model weights from GPU memory dominates the time, and the compute units are underutilized. Verifying K tokens requires loading the weights once (same as generating one token) but performs K times more useful compute. The speedup approaches K × acceptance_rate, where acceptance_rate depends on how well the draft model approximates the target. **Variants and Extensions** - **Self-Speculative Decoding**: The target model itself generates drafts using early exit (partial layers) or a smaller subset of its parameters, eliminating the need for a separate draft model. - **Medusa**: Adds multiple prediction heads to the target model, each predicting tokens at different future positions. A tree-structured verification scheme evaluates multiple candidate sequences in a single forward pass. - **EAGLE**: Uses a lightweight feature-level draft model that operates on the target model's hidden states rather than token embeddings, achieving higher acceptance rates. - **Lookahead Decoding**: Generates N-gram candidates from Jacobi iteration trajectories without requiring a draft model at all. Speculative Decoding is **the key insight that LLM inference wastes most of its computational capacity generating one token at a time** — and that parallel verification is essentially free, converting wasted compute into real throughput gains.