mistral

Mistral is an efficient open-source language model family featuring innovations like sliding window attention. **Company**: Mistral AI (French startup, founded by ex-DeepMind/Meta researchers). **Mistral 7B (Sept 2023)**: Outperformed LLaMA 2 13B despite being half the size. Best 7B model at release. **Key innovations**: **Sliding window attention**: Attend to only recent W tokens (4096), reducing memory, enabling long sequences. **Grouped Query Attention**: Efficient KV cache like LLaMA 2 70B. **Rolling buffer cache**: Fixed memory for KV cache regardless of sequence length. **Architecture**: 32 layers, 4096 hidden dim, 32 heads, 8 KV heads. **Training**: Undisclosed data and process, focused on quality and efficiency. **License**: Apache 2.0 (fully open, commercial OK). **Mixtral 8x7B**: Mixture of Experts version, 46.7B total but 12.9B active per token. Matches GPT-3.5 quality. **Ecosystem**: Widely adopted for fine-tuning, local deployment, and production use. **Impact**: Proved smaller, well-trained models can exceed larger ones. Efficiency-focused approach influential.