mpt

**MPT: Mosaic Pretrained Transformer** **Overview** MPT is a series of open-source LLMs created by **MosaicML** (acquired by Databricks). They were designed to showcase Mosaic's efficient training infrastructure. **Key Innovations** **1. ALiBi (Attention with Linear Biases)** MPT does not use standard Positional Embeddings. It uses ALiBi. - **Benefit**: The model can extrapolate to context lengths *longer* than it was trained on. - MPT-7B-StoryWriter could handle **65k context length** (massive for early 2023) on consumer GPUs. **2. Training Efficiency** MPT was trained from scratch in roughly 9 days for $200k. It demonstrated that training "foundational models" was within reach of startups, not just Google/OpenAI. **3. Commercial License** MPT-7B released with an Apache 2.0 license immediately, allowing commercial use (unlike LLaMA 1 which was research only). **Models** - **MPT-7B**: Base model. - **MPT-30B**: Higher quality, rivals GPT-3. **Legacy** MPT pushed the industry toward longer context windows and faster attention mechanisms (FlashAttention integration).