speculative

**Speculative Decoding for LLM Inference** is **an inference acceleration technique where a smaller, faster model generates candidate tokens speculatively while a larger model verifies them in parallel — eliminating latency bottlenecks through efficient utilization of available compute**. Speculative Decoding addresses a fundamental inefficiency in large language model inference: autoregressive generation requires multiple serial forward passes through the model, and latency-bound inference is the bottleneck. Each token generation requires a forward pass through the entire model, creating a sequential dependency that prevents parallelization despite abundant compute availability. Speculative Decoding leverages the insight that smaller models can generate plausible continuations quickly, and a larger model can verify multiple proposed tokens through a single forward pass. The draft model (smaller, faster) generates k candidate tokens sequentially. The target model (larger, more accurate) runs a single forward pass evaluating all draft tokens and one additional token in parallel. The target model verifies which draft tokens it agrees with — tokens matching the target distribution are accepted, remaining branches are rejected, and generation continues. This approach is efficient because most operations happen in parallel in the target model. Token acceptance rates depend on draft model quality — poor drafts have low acceptance, wasting compute. Well-tuned draft models accept 60-80% of tokens. The speedup is substantial — 1.5-2x speedup is common with carefully tuned draft models. The technique requires no modifications to the target model or tokenizer. Different variants use different draft models — distilled small models, earlier layers of the same model, or even retrieval-based token suggestions. Hardware efficiency improves significantly because the expensive target model forward pass processes multiple positions in parallel rather than single tokens sequentially. Speculative decoding is compatible with other optimization techniques like quantization and batching. The approach works for both greedy decoding and sampling, though sampling requires more complex acceptance criteria. Research shows that the ideal draft model size is task-dependent — too small and acceptance rates drop, too large and generation becomes latency-bound. Hybrid approaches use different draft models for different layers or dynamically adjust draft model complexity. **Speculative decoding dramatically improves language model inference efficiency by enabling parallel token verification, effectively converting sequential token generation into mostly parallel computation.**