speculative decoding llm

**Speculative Decoding** is the **LLM inference acceleration technique that uses a small, fast "draft" model to generate multiple candidate tokens in parallel, which the large "target" model then verifies in a single forward pass — achieving 2-3x speedup with mathematically guaranteed identical output distribution to standard autoregressive generation from the target model alone**. **Why Standard LLM Inference Is Slow** Autoregressive generation is inherently sequential: each token depends on all previous tokens, so the model performs one forward pass per token. For large models (70B+ parameters), each forward pass takes 50-200ms, and most of that time is spent loading model weights from memory (memory-bandwidth-bound). The GPU's compute units are severely underutilized — generating one token at a time wastes the massive parallelism GPUs provide. **How Speculative Decoding Works** 1. **Draft**: A small model (e.g., 1-7B parameters) generates K candidate tokens autoregressively (fast, since the model is small). These K tokens represent a speculative continuation. 2. **Verify**: The large target model processes the entire draft sequence in a single forward pass (just like processing a prompt — fully parallel). It computes the probability distribution at each position. 3. **Accept/Reject**: Starting from the first draft token, each is accepted if the target model's probability for that token is sufficiently high relative to the draft model's probability. A modified rejection sampling scheme ensures the accepted tokens follow exactly the target model's distribution. The first rejected token is resampled from an adjusted distribution. 4. **Repeat**: The process continues from the last accepted token. **Why It Produces Identical Outputs** The acceptance criterion uses a specific probability ratio: accept token x with probability min(1, p_target(x) / p_draft(x)). If rejected, sample from the residual distribution (p_target - p_draft), normalized. This is mathematically proven to reproduce the exact target distribution — there is zero quality degradation. **Speedup Analysis** If the draft model agrees with the target model on ~70% of tokens (common for well-chosen draft/target pairs), and draft length K=5, the expected accepted tokens per verification is ~3.5. Since verification costs roughly the same as generating one token (both are one forward pass), the effective speedup is ~3.5x. **Variants** - **Self-Speculative Decoding**: Uses early exit from the target model itself (e.g., output from layer 8 of a 32-layer model) as the draft, eliminating the need for a separate draft model. - **Medusa**: Adds multiple parallel prediction heads to the target model, each predicting a different future token position. No separate draft model needed. - **EAGLE**: Uses a lightweight autoregressive head on top of the target model's hidden states for more accurate drafting. - **Lookahead Decoding**: Generates multiple n-gram candidates in parallel using Jacobi iteration, verifying them in a single forward pass. Speculative Decoding is **the free lunch of LLM inference** — achieving substantial speedup with zero quality loss by exploiting the asymmetry between sequential generation cost and parallel verification cost.