llemma

**Llemma** is a **34-billion parameter open-source mathematics language model fine-tuned from Code Llama on mathematical texts, competition problems, and formal proofs**, representing the first open-source model demonstrating frontier mathematical reasoning and proof-retrieval capability on university-level mathematics at a scale matching proprietary systems like GPT-4. **Code + Math Fusion** Llemma combines two fundamental insights: | Foundation | Source | Benefit | |-----------|--------|---------| | Code Llama 34B | Meta AI's code specialist | Code understanding improves math (symbolic manipulation) | | Mathematical Data | arXiv, MATH dataset, proofs | Domain-specific reasoning enhancement | Llemma fine-tunes the already code-competent Code Llama on **mathematical texts and formal proofs**—recognizing that mathematics is symbolic computation similar to programming. **Proof Retrieval & Generation**: Unique capability to retrieve and generate **formal mathematical proofs**—not just answers but rigorous derivations. This bridges neural LLMs (pattern matching) with symbolic mathematics (rigorous reasoning). **Performance**: Achieves **47.3% on MATH (university-level competition problems)**—competitive with GPT-3.5 and matching proprietary systems. First fully open model at this level. **Tools Integration**: Designed to pair with symbolic math tools (SageMath, Mathematica)—enabling hybrid workflow where LLM handles reasoning and symbolic systems provide verification. **Legacy**: Proves that **open-source mathematics specialists can reach frontier capability**—democratizing access to advanced mathematical reasoning and enabling researchers to study how LLMs understand formal proofs.