Whole Function Generation is the AI task of generating a complete, correct function implementation given only a natural language docstring and function signature — the primary benchmark task for evaluating code generation models, standardized through OpenAI's HumanEval and Google's MBPP datasets, which measure whether models can translate problem descriptions into working code that passes all unit tests on the first attempt (pass@1) or within k attempts (pass@k).

What Is Whole Function Generation?

The task is precisely scoped: given the function signature and a natural language description of the expected behavior, generate a complete function body:

• Input: def two_sum(nums: List[int], target: int) -> List[int]: with docstring "Return indices of two numbers that add up to target."
• Output: A complete, correct Python implementation using a hash map or two-pointer approach that passes all edge cases.
• Evaluation: The generated function is executed against a hidden test suite. Pass@1 measures whether the first generated solution passes all tests.

Why Whole Function Generation Matters

• Benchmark Standard: HumanEval (164 problems) and MBPP (374 problems) are the canonical benchmarks for comparing code generation models — every major model release (GPT-4, Claude, Gemini, Code Llama, StarCoder) reports pass@1 scores on these datasets.
• End-to-End Correctness: Context-aware completion requires only local coherence (the next line makes sense). Whole function generation requires global correctness — the complete implementation must handle all edge cases, use proper algorithmic complexity, and produce exactly the specified outputs for all inputs.
• Developer Time Compression: The most time-consuming coding subtask is translating a mental model of an algorithm into correct code. When models can reliably generate correct implementations from natural language descriptions, the developer workflow focuses exclusively on problem specification rather than implementation.
• Test-Driven Amplifier: Whole function generation is the computational engine behind AI-assisted TDD — the developer writes the test cases first, the model generates the implementation, and the developer reviews the generated code rather than writing it.

Evaluation Methodology

Pass@k Metric: The statistically unbiased estimator computes pass@k by generating n samples and counting c correct ones:

pass@k = 1 - C(n-c, k) / C(n, k)

This avoids inflating scores by sampling many solutions and reporting the best.

HumanEval Benchmark: 164 hand-written Python programming problems covering algorithms, string manipulation, mathematics, and data structures. Each problem has 7.7 test cases on average. Key milestone scores:

• Original Codex (code-davinci-002): 28.8% pass@1
• GPT-3.5: 48.1% pass@1
• Code Llama 34B Python: 53.7% pass@1
• GPT-4: 67.0% pass@1 (HumanEval)
• Claude 3.5 Sonnet: 92.0% pass@1 (HumanEval, 2024)

Beyond HumanEval: Newer benchmarks address HumanEval's limitations:

• SWE-bench: Real GitHub issues requiring multi-file repository changes, not isolated function generation.
• MBPP: Crowdsourced programming problems with more variety than HumanEval.
• LiveCodeBench: Continuously updated with new problems to prevent contamination.
• EvalPlus: Augmented HumanEval/MBPP with 80x more test cases to catch solutions that pass the original tests by luck.

Current State of the Art

Modern frontier models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) achieve 85-95% pass@1 on HumanEval — effectively saturating the benchmark. The field has shifted to harder benchmarks (SWE-bench Lite: fixing real GitHub bugs) where current best models achieve 40-50%, indicating substantial room for improvement on complex, real-world programming tasks.

Whole Function Generation is the litmus test for code AI capability — the task that cleanly quantifies whether a model can translate human intent into working software, serving as the primary benchmark driving progress in AI-assisted programming research.