tinyllama
**TinyLlama** is a **1.1 billion parameter language model trained on 3 trillion tokens, demonstrating that scaling laws apply even at the smallest feasible scale and creating the largest openly available sub-2B model** — proving that models under 10GB can achieve surprising logical capabilities when trained on enormous token counts, enabling practical deployment on mobile phones, retro hardware, and ultra-low-power devices while maintaining instruction-following and reasoning competence.
**Revolutionary Training Scale**
| Aspect | TinyLlama | Traditional Scaling |
|--------|-----------|-------------------|
| **Parameters** | 1.1B | Typically 7B+ considered viable minimum |
| **Training Tokens** | 3 trillion | Usually 300B-1.4T for small models |
| **Token/Param Ratio** | 2700x | Aggressive overtraining for size |
| **Result** | Surprising reasoning ability | Size constraints limiting capability |
**Design Philosophy**: Rather than normal token budgets, TinyLlama uses extreme overtraining (3 trillion tokens for 1.1B parameters) to compensate for limited parameter capacity—proving scaling laws hold across extreme ranges.
**Accessibility**: At 1.1B parameters, TinyLlama fits in under 2GB of memory. It runs on resourced-constrained devices: iPhones, Raspberry Pi, edge inferenceclusters, and retro computers—democratizing AI access beyond cloud-dependent systems.
**Legacy**: Established that **scale and capability are separable concerns**—smarter training (more tokens) can compensate for smaller models, enabling practical AI everywhere.