codellama
**Code Llama** is a **family of large language models released by Meta AI that are specifically fine-tuned from Llama 2 for code generation and understanding, featuring specialized variants for Python, instruction-following, and general coding** — with innovations including 100,000-token context windows for ingesting entire repositories, native Fill-in-the-Middle (FIM) capability for IDE-style code completion, and free commercial licensing that made it the foundation for dozens of open-source coding assistants.
---
**Model Family Architecture**
Code Llama extends the Llama 2 base with code-specific training stages:
| Variant | Purpose | Training Focus |
|---------|---------|----------------|
| **Code Llama** | General code generation | 500B tokens of code-heavy data on top of Llama 2 |
| **Code Llama - Python** | Python specialist | Additional 100B tokens of Python-specific code |
| **Code Llama - Instruct** | Natural language to code | Instruction-tuned for chat-based coding assistance |
Available in **7B, 13B, 34B, and 70B** parameter sizes, allowing deployment from edge devices to data center GPUs.
---
**Key Technical Innovations**
**Long Context Fine-Tuning (LCFT)**: Code Llama extends the base Llama 2 context from 4,096 to **100,000 tokens** using a dedicated long-context fine-tuning stage with modified RoPE frequencies. This allows the model to ingest entire codebases rather than single files.
**Fill-in-the-Middle (FIM)**: During training, random spans of code are masked and moved to the end of the sequence. The model learns to predict the missing middle given both left and right context — essential for IDE autocompletion where the cursor is between existing code blocks.
**Infilling Training**: Unlike standard left-to-right generation, Code Llama sees both prefix and suffix simultaneously, enabling it to generate code that is syntactically and logically consistent with surrounding context — a capability standard GPT models lack.
---
**🏗️ Ecosystem & Impact**
Code Llama became the **backbone of the open-source coding assistant ecosystem**:
- **Phind-CodeLlama-34B**: Fine-tuned to exceed GPT-4 on HumanEval
- **WizardCoder**: Applied Evol-Instruct to Code Llama for enhanced reasoning
- **Ollama / LM Studio**: Enabled local Code Llama deployment for offline coding
- **Continue.dev**: Open-source VS Code extension powered by Code Llama
**Performance**: Code Llama 34B achieved **48.8% on HumanEval** (vs GPT-3.5's 48.1%), making it the first open-source model to match the commercial standard for code generation. The Python variant pushed this further to **53.7%**.
**License**: Released under a custom Meta license allowing commercial use for applications with under 700M monthly active users — enabling startups and enterprises to build production coding tools without API dependency.