h3 (hungry hungry hippos)
**H3 (Hungry Hungry Hippos)** is a hybrid deep learning architecture that combines **State Space Model (SSM)** layers with **attention mechanisms** to get the best of both worlds — the **linear-time efficiency** of SSMs for long sequences and the **in-context learning** ability of attention.
**Architecture Design**
- **SSM Layers**: The majority of layers use efficient SSM computation (building on **S4**) to process sequences in **O(N)** time, handling long-range dependencies without the quadratic cost of full attention.
- **Attention Layers**: A small number of standard attention layers are interspersed to provide the model with the ability to perform **precise token-to-token comparisons** — something SSMs struggle with on their own.
- **Two SSM Projections**: H3 uses two SSM-parameterized projections — one acting as a **shift** (moving information along the sequence) and another as a **diagonal linear map** — multiplied together before an output projection.
**Why "Hungry Hungry Hippos"?**
The name is a playful reference to the board game, reflecting how the model's SSM layers "gobble up" long sequences efficiently. The H3 paper (by Dan Fu, Tri Dao, et al.) showed that the architecture could match Transformer performance on language modeling while being significantly faster on long sequences.
**Significance**
- **Bridge to Mamba**: H3 was a critical stepping stone between **S4** and **Mamba**. It demonstrated that SSMs needed attention-like capabilities, motivating the development of **selective state spaces** in Mamba.
- **FlashAttention Connection**: H3 was developed by the same research group behind **FlashAttention**, and insights from both projects cross-pollinated.
- **Practical Impact**: Showed that hybrid SSM-attention models could achieve **state-of-the-art** perplexity on language modeling benchmarks while being more efficient than pure Transformers on long sequences.