medusa

Medusa adds parallel prediction heads to language models for speculative token generation, accelerating inference by drafting multiple tokens simultaneously without requiring a separate draft model like in standard speculative decoding. Standard autoregressive: generate one token at a time; each token requires full forward pass; GPU often underutilized (memory-bound). Medusa approach: add extra heads that predict tokens at positions +2, +3, +4, etc. beyond next token; draft multiple future tokens in parallel; verify with single forward pass. Head architecture: lightweight heads (single layer or small MLP) attached to model's last hidden states; each head predicts token at different offset. Verification: original model verifies drafted tokens in one forward pass (parallel verification is cheap); accept prefix of correct tokens, reject incorrect ones. Tree attention: Medusa generates multiple candidate sequences (tree structure); verify entire tree efficiently; increases acceptance rate. Training: fine-tune additional heads on existing model outputs; minimal training cost compared to training draft model. Speed gains: 2-3x speedup depending on acceptance rate and head accuracy; more effective on models with predictable outputs. No draft model: unlike standard speculative decoding, Medusa modifies one model rather than requiring separate models. Medusa demonstrates that simple architectural additions can significantly accelerate inference.