Privacy Budget is the quantitative measure that tracks the cumulative privacy loss of a differential privacy system — expressed as the epsilon (ε) parameter that bounds how much information about any individual can leak through the system's outputs, where each query, training step, or data access consumes a portion of the finite budget, and once exhausted, no further computations can be performed without violating privacy guarantees.

What Is a Privacy Budget?

• Definition: The total amount of privacy loss (ε) that a system is allowed to incur across all operations on a private dataset.
• Core Concept: Every interaction with private data leaks some information — the privacy budget sets a hard limit on total leakage.
• Key Parameter: Epsilon (ε) — lower values mean stronger privacy (ε=0.1 is very strong, ε=10 is weak).
• Finite Resource: Unlike computational budgets that can be replenished, privacy budget is a one-way ratchet — once spent, protection is permanently reduced.

Why Privacy Budget Matters

• Accountability: Provides a concrete, measurable limit on how much privacy can be lost.
• Resource Management: Forces organizations to prioritize which analyses and models are worth the privacy cost.
• Regulatory Compliance: Enables demonstrable compliance with privacy regulations through quantifiable guarantees.
• Composition Control: Without budget tracking, repeated queries could cumulatively destroy privacy.
• Trust Building: Users can be assured their data is protected up to a specified, auditable level.

How Privacy Budget Works

Composition Theorems

• Basic Composition: For k queries each with privacy ε_i, total privacy is Σε_i (linear — pessimistic).
• Advanced Composition: For k queries each with privacy ε, total is O(ε√(k·log(1/δ))) (sublinear — tighter).
• Rényi Composition: Uses Rényi divergence for even tighter privacy accounting.
• Moments Accountant: Numerical tracking providing the tightest known composition bounds for DP-SGD.

Budget Allocation Strategies

• Equal Allocation: Divide budget equally across anticipated queries.
• Priority-Based: Allocate more budget to high-value analyses, less to exploratory queries.
• Adaptive: Dynamically allocate budget based on query importance and remaining balance.
• Hierarchical: Set organizational budget, then sub-allocate to teams and projects.

Practical Considerations

• Setting ε: No universal "right" value — depends on data sensitivity, threat model, and utility requirements.
• Apple: Uses ε=2-8 for local differential privacy in iOS analytics.
• Google: Uses ε=0.5-8 for RAPPOR and Chrome data collection.
• US Census: Used ε≈19.61 for 2020 Census disclosure avoidance.

Privacy Budget is the fundamental resource that makes differential privacy practical — providing the accounting framework that transforms abstract privacy guarantees into concrete, manageable limits that organizations can allocate, track, and audit across all operations on sensitive data.