Falcon LLM (TII)

Overview Falcon is a family of open-source Large Language Models developed by the Technology Innovation Institute (TII) in Abu Dhabi. Upon release (May 2023), Falcon-40B was the top-ranked open-source model on the Hugging Face Open LLM Leaderboard.

Key Models

• Falcon-180B: A massive model comparable to GPT-3.5/GPT-4 in some benchmarks. Released Sept 2023.
• Falcon-40B: The original flagship.
• Falcon-7B: Lightweight version.

Architecture Falcon introduced Multi-Query Attention (MQA) to the mainstream.

• Standard Attention: One head per query/key/value.
• MQA: Shared keys/values across all heads.
• Result: Significantly faster inference and lower memory usage (KV Cache) during text generation.

Data Quality: RefinedWeb TII claimed the secret sauce was their RefinedWeb dataset. They filtered CommonCrawl aggressively, proving (like Dolly) that clean data beats "more data."

Licensing Originally restrictive, TII re-licensed Falcon under Apache 2.0 (fully open metric) due to community feedback, setting a precedent for Open Weights.