starcoder

StarCoder is a family of open-source code generation models developed by the BigCode project (a collaboration between Hugging Face and ServiceNow), trained on The Stack — a large, ethically sourced dataset of permissively licensed code from GitHub. StarCoder represents a commitment to open, transparent, and responsible development of code AI, with full disclosure of training data, model architecture, and evaluation results. The original StarCoder (15.5B parameters) was trained on 80+ programming languages from The Stack v1 (6.4 TB of permissively licensed code), with a context window of 8,192 tokens using multi-query attention for efficient inference. StarCoder2 (2024) expanded the family to three sizes (3B, 7B, 15B parameters) trained on The Stack v2 (67.5 TB from Software Heritage — 4× larger and more diverse than v1), including code, documentation, GitHub issues, Jupyter notebooks, and other code-adjacent natural language content. Key features include: fill-in-the-middle capability (generating code to insert between prefix and suffix — essential for IDE integration), multi-language proficiency (strong performance across Python, JavaScript, Java, C++, and dozens of other languages), long context understanding (StarCoder2 supports 16K+ context windows), and technical chat capability (answering programming questions through instruction-tuned variants like StarChat). StarCoder models achieve competitive performance on HumanEval and MBPP benchmarks, with StarCoder2-15B matching or exceeding larger proprietary models on many code tasks. The project emphasizes ethical training data practices: an opt-out mechanism allows developers to remove their code from training data, and all training data is permissively licensed (Apache-2.0, MIT, BSD). StarCoder powers various open-source coding assistants and can be fine-tuned on domain-specific codebases for specialized applications.