falcon

Falcon is a family of open-source large language models developed by the Technology Innovation Institute (TII) in Abu Dhabi, notable for their high performance achieved through meticulous training data curation rather than novel architecture innovations. The Falcon family includes models at multiple scales: Falcon-7B, Falcon-40B (both released in 2023), and Falcon-180B (2023, one of the largest openly available models at that time). Falcon's key differentiator is its training data — RefinedWeb, a massive dataset created by carefully filtering and deduplicating Common Crawl web data using extensive quality heuristics. RefinedWeb demonstrated that properly filtered web data alone can produce models competitive with those trained on curated multi-source datasets, challenging the assumption that high-quality training requires carefully assembled mixtures of books, academic papers, and specialized corpora. The filtering pipeline includes: URL-based filtering, document-level quality classification, exact and near-deduplication (using MinHash for fuzzy matching), and language identification. Falcon-40B was trained on 1 trillion tokens from RefinedWeb plus curated sources, using a decoder-only transformer architecture with multi-query attention (reducing KV-cache memory requirements) and FlashAttention for efficient training. Upon release, Falcon-40B topped the Open LLM Leaderboard on Hugging Face, outperforming LLaMA and other open models on multiple benchmarks. Falcon-180B (trained on 3.5 trillion tokens) achieved performance between GPT-3.5 and GPT-4 on many tasks. Falcon models were released under the Apache 2.0 license (after initially using a custom license), making them fully open for commercial and research use. The Falcon project's impact extended beyond the models themselves — the RefinedWeb methodology influenced subsequent training data preparation approaches, and TII's investment demonstrated that well-funded non-US organizations could produce competitive open-source foundation models.