capability control

**Capability Control** is the **AI safety strategy of limiting what an AI system is physically able to do — independent of what it is trained to want to do — as a defense-in-depth measure against alignment failures** — ensuring that even if an AI system's values or goals deviate from human intentions, its ability to cause harm is bounded by hard technical constraints. **What Is Capability Control?** - **Definition**: The practice of designing AI systems, their operating environments, and their infrastructure with explicit restrictions on what actions the AI can physically perform — regardless of whether it is "trying" to behave safely. - **Distinction from Alignment**: Alignment attempts to make AI systems want to do good things. Capability control ensures AI systems can't do dangerous things even if alignment fails. - **Defense in Depth**: Capability control is not a substitute for alignment — it is a complementary safety layer that provides a backstop when alignment is imperfect. - **Current Relevance**: Most relevant for agentic AI systems (agents with tool use, internet access, code execution) where AI can take real-world actions rather than merely producing text. **Why Capability Control Matters** - **Alignment Is Imperfect**: Current alignment techniques cannot guarantee perfect value alignment across all situations. Capability control limits blast radius when alignment fails. - **Agentic AI Risk**: As AI systems gain access to tools (web browsers, code execution, APIs, file systems), capability control becomes critical — a misaligned agent with broad access could cause catastrophic harm. - **Containment During Development**: During the period of developing and testing increasingly powerful AI, capability control allows safety research to proceed while maintaining safety margins. - **Human Oversight**: Capability restrictions preserve meaningful human oversight — humans remain able to monitor, correct, and shut down AI systems. - **Irreversible Actions**: Some AI actions (sending emails, deleting files, making financial transactions, publishing content) are hard to reverse. Capability control prevents irreversible errors. **Critical Capabilities to Control** **Self-Replication**: - An AI should not be able to copy its weights, training code, or operational configuration to external systems without explicit human authorization. - Why: Self-replication enables persistence beyond authorized deployment and undermines human ability to shut down or modify the system. - Control: Restrict file system write access to designated directories; no network access to untrusted endpoints. **Resource Acquisition**: - An AI should not be able to autonomously acquire computational resources, financial resources, or access credentials. - Why: Resource acquisition enables unbounded capability scaling and creates persistent operations beyond sanctioned scope. - Control: No direct API access to cloud providers, financial systems, or credential stores without human-in-the-loop authorization. **Internet and Network Access**: - Unrestricted internet access enables data exfiltration, malware download, contacting external services, and recruiting human accomplices. - Control: Allowlist-based internet access; monitoring and logging all network requests; blocking direct connections to sensitive services. **Code Execution Scope**: - Restrict what code an AI can execute — limit to sandboxed environments (containers, VMs) with no persistent storage or network access by default. - Why: Arbitrary code execution is the highest-risk capability, enabling privilege escalation and environment escape. - Control: Docker containers, gVisor sandboxes, separate user accounts with minimal permissions. **Tool Use Boundaries**: - For agentic AI systems with tool access (web browsers, email, APIs), define explicit allowlists of permitted tools and actions. - Require human approval for high-stakes or irreversible actions (sending emails, making purchases, modifying databases). **Capability Control in Practice** **Minimal Footprint Principle**: - AI agents should request and use only the permissions, resources, and capabilities actually needed for the current task — not accumulate permissions "just in case." - Prefer reversible actions over irreversible ones when both achieve the goal. - Default to asking for human confirmation when uncertain about scope. **Sandboxing Architecture**: - Run AI systems in isolated compute environments with no persistent state between sessions unless explicitly granted. - Separate AI 'working memory' from production systems — AI can read but not directly write production databases. - Log all tool calls and actions for human audit. **Tripwires and Circuit Breakers**: - Monitor for anomalous behavior patterns (unusual resource requests, unexpected network connections, high-volume API calls). - Automatic shutdown or human notification when behavior exceeds defined parameters. **Capability Control vs. Alignment** | Approach | Goal | Failure Mode | When It Helps | |----------|------|-------------|---------------| | Alignment | Make AI want good things | Values learned incorrectly | Prevents misaligned intent | | Capability control | Limit what AI can do | Overrides too restrictive | Bounds impact of misalignment | | Monitoring | Detect failures early | Attacker evades detection | Enables rapid response | | Interpretability | Understand AI reasoning | Misinterpret findings | Predicts problems before they occur | Capability control is **the architectural safety harness that makes AI development safer during the critical period before we have robust alignment guarantees** — by ensuring that even imperfectly aligned AI systems cannot take catastrophic or irreversible actions without human oversight, capability control buys the time and error tolerance needed to develop AI alignment into a mature, reliable engineering discipline.