hyena
**Hyena** is a **subquadratic attention replacement that combines long convolutions (computed via FFT) with element-wise data-dependent gating** — achieving O(n log n) complexity instead of attention's O(n²) while maintaining the data-dependent processing crucial for language understanding, matching transformer quality on language modeling at 1-2B parameter scale with 100× speedup on 64K-token contexts, representing a fundamentally different architectural path beyond the attention mechanism.
**What Is Hyena?**
- **Definition**: A sequence modeling operator (Poli et al., 2023) that replaces the attention mechanism with a composition of long implicit convolutions (parameterized by small neural networks, computed via FFT) and element-wise multiplicative gating that conditions processing on the input data — achieving the "data-dependent" property of attention without the quadratic cost.
- **The Motivation**: Attention is O(n²) in sequence length, and all efficient attention variants (FlashAttention, sparse attention, linear attention) are either still quadratic in FLOPs, approximate, or lose quality. Hyena asks: can we build a fundamentally subquadratic operator that matches attention quality?
- **The Answer**: Long convolutions provide global receptive fields in O(n log n) via FFT, and data-dependent gating provides the input-conditional processing that makes attention so powerful. The combination achieves both.
**The Hyena Operator**
| Component | Function | Analogy to Attention |
|-----------|---------|---------------------|
| **Implicit Convolution Filters** | Parameterize convolution kernels with small neural networks, apply via FFT | Like the attention pattern (which tokens interact) |
| **Data-Dependent Gating** | Element-wise multiplication gated by the input | Like attention weights being conditioned on Q and K |
| **FFT Computation** | Convolution in frequency domain: O(n log n) | Replaces the O(n²) QK^T attention matrix |
**Hyena computation**: h = (v ⊙ filter₁(x)) ⊙ (x ⊙ filter₂(x))
Where ⊙ is element-wise multiplication and filters are implicitly parameterized.
**Complexity Comparison**
| Operator | Complexity | Data-Dependent? | Global Receptive Field? | Exact? |
|----------|-----------|----------------|------------------------|--------|
| **Full Attention** | O(n²) | Yes (QK^T) | Yes | Yes |
| **FlashAttention** | O(n²) FLOPs, O(n) memory | Yes | Yes | Yes |
| **Linear Attention** | O(n) | Approximate | Yes (kernel approx) | No |
| **Hyena** | O(n log n) | Yes (gating) | Yes (FFT convolution) | N/A (different operator) |
| **S4/Mamba** | O(n) or O(n log n) | Yes (selective) | Yes (SSM) | N/A (different operator) |
| **Local Attention** | O(n × w) | Yes | No (window only) | Yes (within window) |
**Benchmark Results**
| Benchmark | Transformer (baseline) | Hyena | Notes |
|-----------|----------------------|-------|-------|
| **WikiText-103 (perplexity)** | 18.7 (GPT-2 scale) | 18.9 | Within 1% quality |
| **The Pile (perplexity)** | Comparable | Comparable at 1-2B scale | Matches at moderate scale |
| **Long-range Arena** | Baseline | Competitive | Synthetic long-range benchmarks |
| **Speed (64K context)** | 1× (with FlashAttention) | ~100× faster | Dominant advantage at long contexts |
**Hyena vs Related Subquadratic Architectures**
| Model | Core Mechanism | Complexity | Maturity |
|-------|---------------|-----------|----------|
| **Hyena** | Implicit convolution + gating | O(n log n) | Research (2023) |
| **Mamba (S6)** | Selective State Space Model + hardware-aware scan | O(n) | Production-ready (2024) |
| **RWKV** | Linear attention + recurrence | O(n) | Open-source, active community |
| **RetNet** | Retention mechanism (parallel + recurrent) | O(n) | Research (Microsoft) |
**Hyena represents a fundamentally new approach to sequence modeling beyond attention** — replacing the O(n²) attention matrix with O(n log n) FFT-based implicit convolutions and data-dependent gating, matching transformer quality at moderate scale while delivering 100× speedups on long contexts, demonstrating that the attention mechanism may not be the only path to high-quality language understanding and opening the door to sub-quadratic foundation models.