speculative decoding

**Speculative Decoding** is **an inference acceleration technique where a small draft model rapidly generates multiple candidate tokens, which a large model verifies in batch — achieving 2-4x speedup for large language models without changing outputs through acceptance/rejection sampling**. **Core Algorithm:** - **Draft Model Generation**: small, fast model (e.g., 1B parameters) predicts γ tokens ahead (γ=3-5 typical) in single forward pass — takes 10-20ms on A100 - **Batch Verification**: large model (e.g., 70B Llama) verifies all γ candidate tokens simultaneously in one forward pass — computes attention over draft sequence - **Token Acceptance**: comparing large model logits P_large(x_i) with draft logits P_draft(x_i), accept token if P_large(x_i) > P_draft(x_i) with probability adjustment — maintains exact output distribution - **Rejection Sampling**: if token rejected, resampling from adjusted distribution P_new(x) = max(0, P_large(x) - P_draft(x)) / (1 - P_draft(x)) — preserves correctness **Speedup Mechanism:** - **Latency Reduction**: expected speedup γ_accept = Σ[i=1 to γ] P(accept all i) where P(accept_i) ≈ 0.7-0.9 per token — typical speedup 2-3.5x - **Large Model Efficiency**: amortizing one large model call across multiple tokens (similar to batch size γ) — reduces relative overhead of attention computation - **Draft Model Overhead**: small model adds 5-10% latency (10-20ms) but saves 50-100ms from large model — net gain 40-90ms per iteration - **Cache Reuse**: KV cache from large model verification enables streamlined next iteration — minimal redundant computation **Practical Implementation:** - **Model Pairing**: Llama 70B with Llama 7B draft model achieves 3x speedup with <0.1% accuracy change — commercial services deploy this pattern - **Medusa Framework**: leveraging shared Llama backbone with lightweight head predictors (1.2% parameters) — achieves 2.3x speedup over naive decoding - **HuggingFace Integration**: "Assisted Generation" API enabling drop-in replacement with any fine-tuned draft model — compatible with transformers library - **Threshold Tuning**: adjusting acceptance threshold to balance speed (higher threshold = lower acceptance rate) — critical for different quality requirements **Advanced Strategies:** - **Multi-Draft Ensemble**: using 2-3 different draft models and averaging predictions before verification — improves acceptance rate to 0.92-0.95 - **Adaptive Gamma**: dynamically adjusting lookahead tokens γ based on recent acceptance rates (increase if >0.8, decrease if <0.6) — auto-tuning for optimal throughput - **Prefix Sharing**: caching draft model outputs for common prefixes in batch inference — 30-40% reduction in draft model compute - **Tree Attention**: organizing draft proposals in tree structure enabling parallel verification of competing branches — enables 4-6x speedup with multiple valid continuations **Speculative Decoding is transforming inference economics — enabling production deployment of 70B parameter models on limited hardware while maintaining output quality through verification.**