codellama

Code Llama is Meta's family of specialized code generation models built on the Llama 2 foundation, designed for code understanding, generation, completion, and instruction following across multiple programming languages. Released in August 2023, Code Llama was created by further training Llama 2 on code-heavy datasets, resulting in models that significantly outperform the general-purpose Llama 2 on programming tasks while maintaining strong natural language capabilities. The Code Llama family includes three variants at each size (7B, 13B, 34B, and later 70B parameters): Code Llama (base model — trained on code-heavy data with fill-in-the-middle capability for code completion), Code Llama - Instruct (fine-tuned on instruction-following data — optimized for generating code from natural language descriptions and answering programming questions), and Code Llama - Python (additionally trained on Python-heavy data for superior Python code generation). Key training innovations include: long-context fine-tuning (supporting up to 100K token context windows through position interpolation, enabling analysis of large codebases), infilling training (fill-in-the-middle capability where the model generates code to insert between given prefix and suffix — essential for IDE-style code completion), and instruction tuning via RLHF and self-instruct methods. Code Llama achieves strong results on coding benchmarks: the 34B model scores 53.7% on HumanEval (pass@1) and 56.2% on MBPP, competitive with GPT-3.5 on code tasks. The 70B variant further improved these benchmarks. Being open-source (released under a permissive community license), Code Llama is widely used for local code completion, fine-tuning on domain-specific code, research into code understanding, and as a foundation for commercial AI coding tools. Code Llama supports most popular programming languages including Python, JavaScript, Java, C++, C#, TypeScript, Rust, Go, and many others.