execution feedback

Execution feedback is a code AI paradigm where generated code is actually executed, and any resulting errors, outputs, or test results are fed back to the model to iteratively refine and correct the code until it works correctly. This creates a closed-loop system that goes beyond single-pass code generation by incorporating real-world validation into the generation process. The execution feedback loop typically works as follows: the model generates initial code from a specification or prompt, the code is executed in a sandboxed environment, if errors occur (syntax errors, runtime exceptions, incorrect outputs, failed test cases) the error messages and stack traces are appended to the context, and the model generates a corrected version — repeating until the code passes all tests or a maximum iteration count is reached. Key implementations include: CodeAct (using code actions with execution feedback for agent tasks), Reflexion (combining self-reflection with execution results for iterative improvement), OpenAI's Code Interpreter (executing Python in a sandbox and iterating based on outputs), and AlphaCode (generating many candidates and filtering by execution against test cases). Execution feedback dramatically improves code correctness: models that achieve modest pass@1 rates on single-pass generation can achieve much higher success rates with iterative refinement, as many initial errors are minor issues (off-by-one errors, missing imports, incorrect variable names) that are easily fixed given error messages. The approach mirrors how human developers work — writing code, running it, reading errors, and fixing issues iteratively. Technical requirements include: secure sandboxed execution environments (preventing malicious code from causing harm), timeout mechanisms (preventing infinite loops), resource limits (memory, CPU, disk), and context management (efficiently incorporating execution history without exceeding model context windows). Challenges include handling errors that don't produce informative messages, avoiding infinite retry loops, and managing execution costs.