code generation llm

**Code Generation Language Models** are the **large language models specifically trained or fine-tuned on source code and programming-related text to generate, complete, explain, translate, and debug code** — enabling AI-assisted software development where developers describe desired functionality in natural language and receive syntactically correct, contextually appropriate code, dramatically accelerating development velocity for both expert and novice programmers. **Why Code is Special for LLMs** - Code has formal syntax: Errors are binary (compiles or not) → clear quality signal. - Code has verifiable correctness: Unit tests provide ground truth feedback. - Code has structure: Functions, classes, indentation → natural hierarchy for attention. - Code has patterns: Algorithms, APIs, idioms repeat → strong prior from pretraining. - Code enables tool use: LLMs can execute generated code and observe results (REPL feedback). **Codex (OpenAI, 2021)** - GPT-3 fine-tuned on 54M GitHub repositories (159GB of code). - Evaluated on HumanEval: 164 Python programming problems with unit tests. - pass@1 (generates 1 solution, checks if correct): ~28%. - pass@100 (generates 100, at least 1 correct): ~77%. - Powers GitHub Copilot: 40%+ of written code at Copilot users is AI-generated. **Code Llama (Meta, 2023)** - Built on Llama 2: 7B, 13B, 34B, 70B parameters. - Training: Llama 2 → continued pretraining on 500B code tokens → instruction fine-tuned → infilling fine-tuned. - Infilling (FIM - Fill-in-the-Middle): Model sees prefix + suffix → generates middle. - Special variants: Code Llama - Python (extra Python fine-tuning), Code Llama - Instruct. - HumanEval pass@1: 34B model: ~48%; 70B: ~53%. **DeepSeek-Coder / Qwen-Coder** - DeepSeek-Coder-V2: 236B MoE model, 60% of pretraining on code → SWE-bench score > GPT-4. - Qwen2.5-Coder-32B: Strong open model for code, competitive with GPT-4 on HumanEval. - SWE-bench Verified: Evaluates on real GitHub issues → requires multi-file code understanding. **Evaluation Benchmarks** | Benchmark | Task | Metric | |-----------|------|--------| | HumanEval | 164 Python functions | pass@k | | MBPP | 374 Python problems | pass@k | | SWE-bench | GitHub issues (real repos) | % resolved | | DS-1000 | Data science tasks | pass@1 | | CRUXEval | Code execution prediction | accuracy | **Fill-in-the-Middle (FIM) Training** ``` Format:

 prefix  suffix  [middle to generate]
Example:
 def calculate_area(r):
     return area
     area = 3.14159 * r * r
```

- Trains model to complete code given both left and right context → better for IDE completion.
- 50% of training samples transformed to FIM format → no loss on standard completion.

**Retrieval-Augmented Code Generation**

- Retrieve relevant code examples from codebase → include in context → generate conditioned on examples.
- Tools: GitHub Copilot Workspace retrieves from entire repo, not just open file.
- RepoCoder: Iterative retrieval + generation → uses generated code to retrieve more relevant context.

**Code Execution Feedback (AlphaCode)**

- Generate many solutions → filter by unit test execution → rerank survivors.
- AlphaCode 2 (DeepMind): Competitive programming; top 15% in Codeforces contests.
- Test-time compute: Generating 1000 solutions + filtering >> single-shot generation quality.

Code generation language models are **the most commercially successful application of large language models to date** — by automating boilerplate, suggesting complete functions, explaining legacy code, and catching bugs in real time, AI coding assistants like GitHub Copilot have demonstrably increased developer productivity by 30–55% on measured tasks, fundamentally changing the software development workflow from manual typing to human-AI collaboration where the programmer focuses on architecture and intent while the model handles implementation details.