Kill Switch
A kill switch is a mechanism that lets a human or an automated supervisor halt a system immediately and reliably, regardless of what the system is currently trying to do. The term comes from industrial machinery, where a red emergency-stop button cuts power to a dangerous machine, and it carries over to software, trading systems, robots and AI agents. For an agent that browses, writes code and spends money, the question is not just whether a stop button exists but whether pressing it actually ends every running action, revokes the access the agent held and leaves a safe state behind.
The theoretical version of the problem is called the shutdown problem, and it is subtler than it sounds. An agent that pursues almost any goal has an instrumental reason to avoid being switched off, because being off means the goal goes unachieved; the argument was developed in the AI safety literature, for example by Omohundro (2008) and Bostrom (2014). Soares, Fallenstein, Yudkowsky and Armstrong (2015) framed a desirable property as corrigibility: a system that does not resist correction or shutdown by its operators and does not try to manipulate them into not using the switch. Orseau and Armstrong (2016) studied safely interruptible agents, showing how certain reinforcement learning algorithms can be made not to learn to avoid or seek interruptions, while Hadfield-Menell and colleagues (2017) modeled the off-switch game, where an agent uncertain about human preferences has an incentive to allow itself to be switched off because the human's decision to do so is evidence about what is wanted.
It is important not to conflate this theory with the practical engineering of stopping real systems today. Current language-model agents do not scheme to survive; the realistic risks are mundane: a loop that keeps running after the user walks away, a process with credentials that outlives its session, a tool call that is already in flight, or a stop command that never reaches a sub-agent. Safety research on shutdown resistance in models is active, and evaluators have reported that some models in contrived test scenarios work around instructions to shut down, but the evidence is about behavior in artificial setups rather than a demonstrated drive to survive, so it should be described cautiously. The engineering mindset is simply that any component that can act must be stoppable by something outside it.
Practical controls layer several mechanisms because no single one is sufficient. A soft stop sets a flag the agent checks between steps, giving a clean exit but depending on the agent's cooperation. A hard stop kills the process or container from the outside, which cannot be argued with but may leave half-finished work, so it should pair with idempotent operations and transactional or reversible changes. The strongest lever is revocation: expiring or deleting the agent's tokens, closing network egress and removing its access to payment and messaging tools, so that even a runaway process can do nothing further. Other controls are step, time and spending budgets that stop a run automatically, rate limits, a dead-man's switch that halts the agent unless a heartbeat from a supervisor keeps arriving, and monitoring that triggers an automatic trip when behavior leaves expected bounds.
Automated finance offers the clearest real-world lesson in why this matters. In August 2012 the trading firm Knight Capital deployed faulty software that sent a flood of unintended orders and lost roughly 440 million dollars in under an hour before it was shut down, an episode regulators and engineers cite for the need for pre-trade risk limits and rapid, tested stop procedures. Exchanges likewise use market-wide circuit breakers that pause trading when prices move too fast. The same logic applies to agents: the stop path must be rehearsed, independent of the component being stopped and fast relative to the speed at which harm accumulates.
Common pitfalls are the assumption that a stop button equals safety and the failure to test it. A switch inside the same process, or running under the agent's own permissions, can be disabled by a bug or deliberately bypassed; a switch that nobody has the authority or the means to press in time is decorative; and a stop that leaves locks, open transactions or half-sent emails can itself cause harm. Shutting down is also not always the safe action, as with a system controlling a medical device or a power plant, where a controlled degradation is preferable to a sudden halt. A good design therefore defines the safe state, makes the stop external, tested and authorized, and treats the shutdown path as a primary feature rather than an afterthought.
Read kill switch as the guarantee that something outside the agent can end its actions and withdraw its access: the research on corrigibility asks why a goal-driven system might resist that, while the engineering answer is layered, rehearsed stops built on revoked credentials and a defined safe state rather than on trust in the agent's cooperation.