Llemma is a 34-billion parameter open-source mathematics language model fine-tuned from Code Llama on mathematical texts, competition problems, and formal proofs, representing the first open-source model demonstrating frontier mathematical reasoning and proof-retrieval capability on university-level mathematics at a scale matching proprietary systems like GPT-4.

Code + Math Fusion

Llemma combines two fundamental insights:

Llemma fine-tunes the already code-competent Code Llama on mathematical texts and formal proofs—recognizing that mathematics is symbolic computation similar to programming.

Proof Retrieval & Generation: Unique capability to retrieve and generate formal mathematical proofs—not just answers but rigorous derivations. This bridges neural LLMs (pattern matching) with symbolic mathematics (rigorous reasoning).

Performance: Achieves 47.3% on MATH (university-level competition problems)—competitive with GPT-3.5 and matching proprietary systems. First fully open model at this level.

Tools Integration: Designed to pair with symbolic math tools (SageMath, Mathematica)—enabling hybrid workflow where LLM handles reasoning and symbolic systems provide verification.

Legacy: Proves that open-source mathematics specialists can reach frontier capability—democratizing access to advanced mathematical reasoning and enabling researchers to study how LLMs understand formal proofs.