orca

**Orca** is a **13B parameter model from Microsoft Research that solved the "imitation gap" problem in small language models by training on explanation traces rather than just question-answer pairs** — demonstrating that teaching a student model how the teacher thinks (step-by-step reasoning, system instructions) rather than just what the teacher says produces dramatically better reasoning capabilities, with Orca-13B surpassing ChatGPT (GPT-3.5) on complex reasoning benchmarks despite being much smaller. **What Is Orca?** - **Definition**: A research model from Microsoft Research (2023) that fine-tuned LLaMA-13B on 5 million examples of GPT-4's reasoning traces — where each training example includes the system instruction, the question, and GPT-4's detailed step-by-step explanation, not just the final answer. - **The Imitation Problem**: Previous small models (Vicuna, Alpaca) trained on GPT-4 outputs learned to copy the style (fluent, confident responses) but not the substance (actual reasoning ability). They sounded smart but failed on complex reasoning tasks. - **Explanation Tuning**: Orca's key innovation — instead of training on [Question → Answer] pairs, it trains on [System Instruction + Question → Detailed Explanation + Answer] tuples. The system instructions include "Explain your step-by-step reasoning," "Think carefully before answering," and "Show your work." - **Progressive Learning**: Orca first learns from ChatGPT (GPT-3.5) explanations (easier, more examples), then from GPT-4 explanations (harder, higher quality) — a curriculum that progressively builds reasoning capability. **Why Orca Matters** - **Reasoning Breakthrough**: Orca-13B surpassed ChatGPT (GPT-3.5-Turbo) on BigBench-Hard, a benchmark specifically designed to test complex reasoning — proving that small models can reason well when trained on reasoning traces rather than just answers. - **"Data Density" Insight**: Orca demonstrated that it's not about the quantity of training data but the density of reasoning information per example — 5M high-quality explanation traces outperformed datasets with 10× more simple Q&A pairs. - **Influenced the Field**: Orca's explanation tuning approach influenced subsequent models — WizardLM, OpenHermes, and many others adopted the practice of including reasoning traces and system instructions in training data. - **Microsoft Research Contribution**: As a Microsoft Research paper, Orca provided rigorous experimental validation — controlled comparisons showing exactly where explanation tuning improves over standard fine-tuning. **Orca Model Versions** | Model | Base | Training Data | Key Achievement | |-------|------|-------------|----------------| | Orca | LLaMA-13B | 5M GPT-4 explanations | Beat ChatGPT on BigBench-Hard | | Orca 2 | LLaMA-2-7B/13B | Improved explanation data | Better reasoning with smaller base | **Orca is the Microsoft Research model that proved small language models can reason like large ones when taught how to think** — by training on GPT-4's step-by-step explanation traces rather than just final answers, Orca demonstrated that "data density" (reasoning information per example) matters more than data quantity, fundamentally changing how the community approaches small model training.