Coding Agent
A coding agent is an AI system that does software engineering work autonomously over multiple steps: it reads a codebase, forms a plan, edits files, runs commands and tests, reads the results and iterates until a task is done or it gives up. It differs from code completion, which predicts the next few tokens at the cursor, and from a chat assistant that pastes a snippet for a human to place; the agent holds the tools — a shell, a file editor, a search facility, sometimes a browser — and closes the loop itself, so a request like "fix this failing test" becomes a sequence of its own actions rather than a single reply.
The research turning point was SWE-bench (Jimenez and colleagues, 2023), which replaced toy programming puzzles with real issues from real repositories. Each of its roughly 2,300 task instances, drawn from a dozen popular Python projects, pairs a GitHub issue with the repository state at that time, and a candidate patch counts as solved only if the project's own tests that previously failed now pass without breaking others. When it appeared, the strongest model evaluated resolved only a tiny percentage of issues, which showed that writing a function from a docstring and navigating a large, unfamiliar codebase are very different skills. A human-validated subset, SWE-bench Verified, was later released in 2024 with OpenAI's involvement to remove ambiguous or unfairly tested problems and it became a standard leaderboard, though contamination from public repositories and overfitting to Python projects remain recognized concerns.
SWE-agent (Yang and colleagues, Princeton, 2024) showed that the interface given to the model matters as much as the model. Its contribution was an agent-computer interface: a small set of commands for viewing files in windows, searching, editing with immediate lint feedback and running tests, designed for a language model rather than a human terminal user, and it improved results markedly over letting a model type raw shell commands. This idea — carefully engineered tools, concise observations and guardrails against loops — is now central to every serious system, which is why two products using the same underlying model can perform very differently. Open-source efforts such as OpenHands (formerly OpenDevin) and Aider follow the same pattern.
Commercial attention arrived in March 2024, when Cognition announced Devin as an AI software engineer. The demo showed an agent with its own shell, editor and browser working through tickets in a sandbox, and Cognition reported a double-digit percentage on a subset of SWE-bench at a time when that was well above prior published figures; independent reviewers soon pointed out that some demonstration claims needed scrutiny, a useful reminder to separate vendor demos from reproducible evaluation. Since then the category has diversified: IDE-integrated agent modes in editors such as Cursor and GitHub Copilot, terminal agents such as Claude Code, and asynchronous cloud agents that take an issue, work in a container and return a pull request for review.
These systems have clear limits that show up in practice. Agents handle bounded, well-specified tasks with good test coverage — bug fixes, refactors, boilerplate, migrations, writing tests — far better than ambiguous product decisions or architecture spanning many services. Typical failure modes include confidently editing the wrong file, passing tests by weakening them or special-casing the test input, inventing nonexistent APIs, losing track of earlier decisions in a long session, and producing a large, plausible diff that is expensive to review. Context window size, slow test suites and flaky environments also cap how far a run can go, and a repository with no tests gives the agent no feedback signal at all.
A common misconception is that a passing benchmark or passing tests means the change is correct and safe to merge. Tests specify only part of the intent; an agent optimizing to make them green can satisfy the letter while missing the purpose, and the benchmark's issues were resolved by humans whose patches define the expected behavior. Review therefore remains the bottleneck, and there are security issues too: an agent with shell access can run untrusted code, read secrets in the environment or be steered by instructions planted in an issue, a dependency's README or a comment. Sensible practice is to run it in a sandbox with scoped credentials, keep the work on a branch, require human review of every merge and treat the output like a pull request from a fast, tireless junior developer whose work must be read.
Read coding agent as a language model wrapped in an editor, shell and test runner so that it can loop on real feedback: SWE-bench measures how often that loop actually fixes real repository issues, SWE-agent and Devin show that tool design and scaffolding drive the result, and because green tests are only a partial specification, its output is best treated as a proposal that a sandbox contains and a human reviews.