falcon

**Falcon LLM (TII)** **Overview** Falcon is a family of open-source Large Language Models developed by the **Technology Innovation Institute (TII)** in Abu Dhabi. Upon release (May 2023), Falcon-40B was the top-ranked open-source model on the Hugging Face Open LLM Leaderboard. **Key Models** - **Falcon-180B**: A massive model comparable to GPT-3.5/GPT-4 in some benchmarks. Released Sept 2023. - **Falcon-40B**: The original flagship. - **Falcon-7B**: Lightweight version. **Architecture** Falcon introduced **Multi-Query Attention (MQA)** to the mainstream. - Standard Attention: One head per query/key/value. - MQA: Shared keys/values across all heads. - **Result**: Significantly faster inference and lower memory usage (KV Cache) during text generation. **Data Quality: RefinedWeb** TII claimed the secret sauce was their **RefinedWeb** dataset. They filtered CommonCrawl aggressively, proving (like Dolly) that clean data beats "more data." **Licensing** Originally restrictive, TII re-licensed Falcon under Apache 2.0 (fully open metric) due to community feedback, setting a precedent for Open Weights.