speculative decoding llm

**Speculative Decoding** is the **LLM inference acceleration technique that uses a smaller, faster "draft" model to generate candidate token sequences speculatively, then verifies them in a single forward pass of the larger target model — accepting correct tokens and rejecting wrong ones, achieving 2-3x speedup without any change in output quality because the verification ensures the final distribution is mathematically identical to sampling from the target model alone**. **Why Standard Autoregressive Decoding Is Slow** Standard LLM generation produces one token per forward pass. Each forward pass of a 70B-parameter model takes the same time regardless of whether it's computing a predictable function word ("the") or a creative content word. The GPU is underutilized during single-token generation because the computation is memory-bandwidth-bound — the entire model must be read from HBM to compute a single output token. **How Speculative Decoding Works** 1. **Draft Phase**: A small model (1-7B parameters, or a non-autoregressive model) quickly generates K candidate tokens (typically K=4-8). This is fast because the draft model is much smaller. 2. **Verification Phase**: The target model processes all K candidate tokens in a single forward pass (as if they were the prompt continuation). This produces probability distributions at each position. 3. **Acceptance/Rejection**: For each position, the candidate token is accepted with probability min(1, p_target(t)/p_draft(t)). If a token is rejected, it is resampled from a corrected distribution. All tokens after the first rejection are discarded. 4. **Result**: On average, multiple tokens are accepted per verification pass, producing >1 token per large-model forward pass. **Theoretical Guarantee** The acceptance-rejection scheme is designed so the marginal distribution of accepted tokens is exactly p_target. The output is statistically identical to autoregressive sampling from the target model — no quality degradation whatsoever. **Practical Speedup Factors** - **Draft-Target Alignment**: The more similar the draft model's distribution is to the target, the higher the acceptance rate. Models from the same family (e.g., Llama 7B drafting for Llama 70B) have high alignment (acceptance rate 70-85%). - **K (Speculation Length)**: Longer speculation means more potential tokens per verification but lower probability of accepting all K. Optimal K is typically 4-8. - **Batch Size**: At batch size 1, speculative decoding provides 2-3x speedup. At large batch sizes, the target model is already compute-saturated, and speculative decoding provides diminishing returns. **Variants** - **Self-Speculative Decoding**: The target model itself generates drafts using early-exit or layer-skipping, eliminating the need for a separate draft model. - **Medusa**: Adds multiple prediction heads to the target model that predict K future tokens simultaneously. Verification is integrated into the model itself. Speculative Decoding is **the batch-processing hack for autoregressive generation** — exploiting the fact that verifying a sequence is cheaper than generating it one token at a time, converting the sequential bottleneck into a parallel verification step.