incoder

**InCoder** is a **code generation model by Meta AI that pioneered Fill-in-the-Middle (FIM) training, enabling models to predict missing code given both left and right context** — a fundamental capability for IDE code completion where the cursor sits between existing code blocks, trained by randomly masking code spans during pre-training and teaching models to reconstruct missing segments, which became the standard training technique for Code Llama, StarCoder, and virtually every modern code generation model. **The Fill-in-the-Middle Innovation** Standard language models generate text left-to-right. InCoder introduced **bidirectional context awareness** for code by training on masked span prediction: | Approach | Context | Capability | |----------|---------|-----------| | **Standard GPT** | Left context only | Generate only what comes next | | **InCoder FIM** | Left + right context | Fill missing code in the middle | **Technical Innovation**: During pre-training, random code spans are extracted and moved to the end of sequences. The model learns to read both prefix (code before cursor) and suffix (code after cursor) to reconstruct the missing span — enabling IDE autocompletion where developers write non-linearly. **Impact & Legacy**: FIM became arguably the **most influential code training innovation** after transformers. Every major code model adopted it: Code Llama, StarCoder, DeepSeek Coder, Copilot—all use FIM as a core training objective. InCoder proved that **bidirectional reasoning** is essential for practical code completion quality.