MPT: Mosaic Pretrained Transformer

Overview MPT is a series of open-source LLMs created by MosaicML (acquired by Databricks). They were designed to showcase Mosaic's efficient training infrastructure.

Key Innovations

1. ALiBi (Attention with Linear Biases) MPT does not use standard Positional Embeddings. It uses ALiBi.

• Benefit: The model can extrapolate to context lengths longer than it was trained on.
• MPT-7B-StoryWriter could handle 65k context length (massive for early 2023) on consumer GPUs.

2. Training Efficiency MPT was trained from scratch in roughly 9 days for $200k. It demonstrated that training "foundational models" was within reach of startups, not just Google/OpenAI.

3. Commercial License MPT-7B released with an Apache 2.0 license immediately, allowing commercial use (unlike LLaMA 1 which was research only).

Models

• MPT-7B: Base model.
• MPT-30B: Higher quality, rivals GPT-3.

Legacy MPT pushed the industry toward longer context windows and faster attention mechanisms (FlashAttention integration).