Differential Privacy in Machine Learning Formal Guarantees for Data-Driven Models
# Differential Privacy in Machine Learning: Formal Guarantees for Data-Driven Models
## Introduction & Motivation
Machine learning models trained on sensitive data, medical records, financial transactions, private communications, browsing history, carry an inherent risk that the trained model itself leaks information about the individuals whose data contributed to it. This risk is not merely theoretical: a substantial body of research has demonstrated that model outputs, confidence scores, and even raw parameter values can be probed by an adversary to infer whether a specific individual's record was present in the training set (membership inference), to reconstruct approximate versions of training examples (model inversion), or to extract verbatim memorized sequences from language models trained on text corpora. Differential privacy (DP) directly addresses this risk by providing a rigorous, mathematically provable guarantee that bounds precisely how much any single individual's data can influence a computation's output, independent of what auxiliary information an attacker might possess.
Unlike ad hoc anonymization techniques, such as removing directly identifying fields or aggregating records into coarse bins, which have repeatedly been shown to be reversible through auxiliary data linkage attacks, differential privacy offers a composable, worst-case guarantee that holds regardless of what other information an adversary has access to, including information not yet known at the time the private computation was performed. This composability, the fact that the total privacy cost of multiple differentially private computations can be bounded and reasoned about mathematically, is what makes differential privacy suitable as a foundation for training entire machine learning pipelines, rather than a one-off technique applied to a single data release.
Applying differential privacy to machine learning specifically, most prominently through differentially private stochastic gradient descent (DP-SGD), has moved from a largely theoretical research area to a practically deployed technique, used by organizations including Apple, Google, and the U.S. Census Bureau, precisely because it offers the first rigorous, quantifiable answer to the question "how much does this specific individual's data affect this trained model," a question that ad hoc privacy techniques cannot answer with comparable mathematical certainty.
## Core Concepts & Theory
Differential privacy is defined relative to a notion of neighboring datasets: two datasets $D$ and $D'$ are neighbors if they differ in exactly one individual's record (either that record is added, removed, or its value is changed, depending on the specific neighboring-datasets convention adopted). A randomized mechanism $\mathcal{M}$ satisfies $(\epsilon, \delta)$-differential privacy if, for every pair of neighboring datasets and every possible measurable set of outputs, the probability of the mechanism producing an output in that set differs between the two neighboring datasets by at most a multiplicative factor of $e^{\epsilon}$, plus a small additive slack $\delta$. Intuitively, this means an observer of the mechanism's output cannot confidently determine, beyond a bounded statistical margin controlled by $\epsilon$ and $\delta$, whether any specific individual's data was included in the input dataset at all, since the mechanism would have produced a nearly indistinguishable output distribution either way.
The privacy parameter $\epsilon$ (the privacy budget or privacy loss) controls the strength of the guarantee: smaller $\epsilon$ means a tighter bound on how distinguishable neighboring datasets' output distributions can be, corresponding to stronger privacy but, as an unavoidable consequence, requiring more noise or randomness be injected into the computation, which in turn degrades the accuracy or utility of the mechanism's output. The additive term $\delta$, typically set to a value much smaller than the reciprocal of the dataset size, allows for a small probability of a stronger-than-$\epsilon$ privacy violation, which is necessary to obtain tractable mechanisms such as the Gaussian mechanism that would otherwise require unbounded noise to satisfy pure $\epsilon$-DP (with $\delta = 0$) exactly.
Sensitivity, the maximum amount by which a query's true output can change between any two neighboring datasets, is the second core quantity underlying most differentially private mechanisms: a query with low sensitivity requires proportionally less noise to achieve a given privacy level than a query whose output can swing wildly based on a single individual's record. The Laplace mechanism, which adds noise drawn from a Laplace distribution scaled to the query's sensitivity divided by $\epsilon$, achieves pure $\epsilon$-differential privacy for any real-valued query, while the Gaussian mechanism, adding calibrated Gaussian noise, achieves the more permissive $(\epsilon, \delta)$-differential privacy and is the mechanism underlying DP-SGD's per-step noise injection due to its more favorable behavior under repeated (composed) application.
## Mathematical Formulation
Formally, a randomized mechanism $\mathcal{M}: \mathcal{D} o \mathcal{R}$ satisfies $(\epsilon, \delta)$-differential privacy if, for all neighboring datasets $D, D'$ and all measurable subsets $S \subseteq \mathcal{R}$ of possible outputs,
$$ \Pr[\mathcal{M}(D) \in S] \leq e^{\epsilon} \cdot \Pr[\mathcal{M}(D') \in S] + \delta $$
The Laplace mechanism, for a query $f: \mathcal{D} o \mathbb{R}$ with sensitivity $\Delta f = \max_{D, D'} |f(D) - f(D')|$ over neighboring datasets, releases
$$ \mathcal{M}(D) = f(D) + ext{Lap}\left( \frac{\Delta f}{\epsilon} ight) $$
which satisfies pure $\epsilon$-differential privacy, since the ratio of Laplace densities at any output point is bounded exactly by $\exp(\epsilon)$ whenever the true query outputs on neighboring datasets differ by at most $\Delta f$.
DP-SGD modifies standard stochastic gradient descent in two steps applied to each individual example's gradient within a minibatch: first, per-example gradient clipping, bounding each example's contribution to the gradient update by rescaling any gradient whose $\ell_2$ norm exceeds a clipping threshold $C$,
$$ \bar{g}_i = g_i \cdot \min\left(1, \frac{C}{\|g_i\|_2} ight) $$
and second, adding calibrated Gaussian noise to the sum (or average) of the clipped per-example gradients before applying the update,
$$ ilde{g} = \frac{1}{|B|}\left( \sum_{i \in B} \bar{g}_i + \mathcal{N}(0, \sigma^2 C^2 I) ight) $$
where $\sigma$, the noise multiplier, together with the clipping norm $C$, the batch size, the number of training steps, and the composition theorem used to track cumulative privacy loss across steps, jointly determine the total $(\epsilon, \delta)$ privacy guarantee for the entire training run.
Basic composition bounds the total privacy loss of $k$ sequentially applied $(\epsilon_0, \delta_0)$-DP mechanisms as simply additive, $\epsilon_{ ext{total}} = k \epsilon_0$, $\delta_{ ext{total}} = k \delta_0$, while advanced composition offers a tighter bound that scales closer to $O(\sqrt{k})$ rather than $O(k)$ for a modest additional delta cost, and the moments accountant (and its modern successor, Renyi differential privacy accounting) provides tighter bounds still, specifically tailored to the repeated, subsampled Gaussian mechanism structure that DP-SGD exhibits, which is what makes training deep models over many thousands of steps with a usable privacy budget practically feasible at all.
## Advanced Theory & Extensions
Renyi differential privacy (RDP), a relaxation that tracks privacy loss using Renyi divergence rather than the direct $(\epsilon, \delta)$ definition, offers particularly clean, tight composition properties (RDP privacy costs of sequentially composed mechanisms simply add at each Renyi order), and has become the standard internal accounting mechanism inside most practical DP-SGD implementations, since it can be converted back to a standard $(\epsilon, \delta)$-DP guarantee at the end of training via a well-understood conversion formula, combining tight composition tracking with an interpretable final guarantee. Privacy amplification by subsampling further tightens the achievable privacy guarantee, formalizing the intuitive fact that a mechanism applied only to a randomly selected subsample of the data (as DP-SGD does at every minibatch step) leaks strictly less information about any given individual than the same mechanism applied to the full dataset, since an individual's data might not even have been included in a given step's subsample at all.
The relationship between differential privacy and generalization is a particularly deep theoretical connection: any $(\epsilon, \delta)$-DP mechanism provably generalizes, in the sense that its output cannot overfit adaptively to the specific training sample beyond a bound controlled by $\epsilon$, which explains both why DP-trained models tend to close the gap between training and test performance and why differential privacy has found applications in adaptive data analysis and reusable holdout mechanisms entirely independent of any concern about protecting individual privacy. Local differential privacy, a stricter variant in which noise is added to each individual's data before it ever leaves their device or is aggregated (rather than the more common central model, where a trusted curator adds noise to an aggregate computation over raw, unperturbed individual data), trades a substantially worse privacy-utility tradeoff for the elimination of any need to trust a central data curator, and underlies deployed systems such as Apple's differentially private telemetry collection and Google's RAPPOR.
## Computational Considerations
DP-SGD's per-example gradient clipping step is its dominant additional computational cost relative to standard SGD, since computing and clipping gradients on a per-example basis, rather than the aggregate minibatch gradient that standard automatic differentiation frameworks compute natively and far more efficiently, generally requires specialized "vectorized" per-example gradient computation techniques (such as JAX's vmap or specialized PyTorch libraries like Opacus) to avoid an otherwise prohibitive computational and memory overhead from looping over individual examples within a batch. Even with efficient vectorization, per-example gradient computation and clipping typically impose a measurable throughput penalty, commonly in the range of a few times slower than standard SGD, that grows somewhat with model size and per-example gradient memory footprint.
Privacy accounting itself, tracking the cumulative $(\epsilon, \delta)$ cost across potentially hundreds of thousands of training steps using RDP or the moments accountant, is computationally cheap relative to the training itself, typically requiring only a numerical search over Renyi orders to find the tightest achievable conversion to a final $(\epsilon, \delta)$ guarantee given the noise multiplier, sampling rate, and number of steps, and modern DP libraries provide this accounting as an efficient, largely automated utility separate from the training loop itself. Large batch sizes are computationally and statistically favorable for DP-SGD specifically, since larger batches both amortize the fixed noise addition over more per-example gradients (improving the signal-to-noise ratio of the averaged, noised gradient) and benefit more from privacy amplification by subsampling at a fixed overall sampling rate, motivating DP training recipes that favor substantially larger batch sizes than would typically be used for the same model trained without privacy constraints.
## Practical Implementation Strategies
Selecting the clipping norm $C$ requires balancing two competing effects: too small a clipping threshold discards genuine gradient signal from examples whose true gradient exceeds the threshold, systematically biasing the training signal, while too large a threshold requires proportionally more noise to be added (since noise scale is directly proportional to $C$) to achieve the same target privacy level, degrading the signal-to-noise ratio of every update; a common practical approach is to select $C$ near the median or a modest percentile of observed per-example gradient norms from a non-private trial run or an early phase of training, rather than tuning it purely as an abstract hyperparameter. The noise multiplier $\sigma$ is typically the primary lever tuned to hit a target overall privacy budget $\epsilon$ for a given, fixed number of training steps and batch size, using a privacy accounting library to numerically search for the smallest $\sigma$ (and hence best utility) that still achieves the target $\epsilon$ at the end of training.
Model architecture choices interact meaningfully with DP-SGD's noise sensitivity: models with fewer parameters and simpler architectures generally tolerate DP-SGD's added noise better than very large, high-capacity models, since noise is added per-parameter-dimension (via the Gaussian mechanism's covariance structure) and a fixed noise budget spread across more dimensions provides proportionally less signal per dimension for high-dimensional models, which is part of why DP fine-tuning of a smaller number of parameters (such as only the final layers, or low-rank adapters) of an already-pretrained model frequently achieves a substantially better privacy-utility tradeoff than DP training of an entire large model from scratch. Practitioners should validate the achieved privacy guarantee using a well-tested, peer-reviewed accounting library rather than hand-implementing composition tracking, since subtle errors in privacy accounting (miscounting the effective sampling rate, or using basic rather than advanced or RDP-based composition) have historically led to publicly reported overstatements of the actual privacy protection achieved by deployed systems.
## Benchmark Datasets & Evaluation
DP machine learning research commonly reports accuracy on standard benchmarks (MNIST and CIFAR-10 for DP image classification, various NLP benchmarks for DP language model fine-tuning) at a range of fixed privacy budgets $\epsilon$, typically spanning from very strong privacy ($\epsilon$ around 1 to 3) to substantially weaker but more research-conventional privacy levels ($\epsilon$ around 8 or higher), to illustrate the concrete privacy-utility tradeoff curve for a given model architecture and training recipe rather than reporting accuracy at only a single arbitrarily chosen privacy level. Membership inference attack success rate (measured as attack accuracy or AUC in distinguishing training members from non-members) is an increasingly standard empirical complement to the formal $(\epsilon, \delta)$ guarantee, since it offers a concrete, attack-grounded sanity check that a claimed privacy guarantee corresponds to genuinely reduced practical vulnerability, and discrepancies between very loose formal guarantees (large $\epsilon$) and empirically low measured attack success are common and worth reporting alongside each other.
The U.S. Census Bureau's adoption of differential privacy for the 2020 Census, and the associated public "TopDown algorithm" and its documented privacy-accuracy tradeoff analysis across different geographic aggregation levels, has become a widely cited real-world case study and benchmark for evaluating differentially private algorithms' practical accuracy tradeoffs at population statistics scale, distinct from but complementary to machine-learning-specific benchmarks. Privacy budget accounting itself, verifying that a claimed $(\epsilon, \delta)$ value is correctly derived given the mechanism's parameters, is increasingly treated as a first-class evaluation artifact, with tools such as Google's DP accounting library and Opacus's built-in RDP accountant used as reference implementations against which custom accounting code should be validated.
## Key Challenges & Limitations
The privacy-utility tradeoff remains differential privacy's most significant practical limitation: achieving strong privacy guarantees (small $\epsilon$) on complex, high-dimensional models frequently incurs substantial accuracy degradation relative to non-private training, and this degradation tends to worsen as model size and dataset dimensionality increase, meaning DP training of very large modern deep learning models at strong privacy levels remains considerably more accuracy-costly than DP training of smaller, simpler models, an active and unresolved tension for applying DP to frontier-scale models. Fairness concerns compound this tradeoff: DP-SGD's noise and clipping have been empirically observed to disproportionately degrade accuracy on minority subgroups and rare classes within a dataset, since these examples, being underrepresented, are more sensitive to both gradient clipping (which can systematically suppress their comparatively larger, more atypical gradients) and to noise (which provides a worse signal-to-noise ratio for the sparser statistical signal these subgroups contribute).
Correctly setting and interpreting $\epsilon$ in a way that is meaningful to non-expert stakeholders remains a persistent challenge, since the numeric value of $\epsilon$ does not translate into an intuitive, universally agreed-upon notion of "how private" a system is, and organizations have adopted widely varying $\epsilon$ values (from below 1 to well over 10) for superficially similar applications, complicating both public communication and cross-organizational comparison of privacy claims. Differential privacy also only protects against the specific threat model it formally defines, membership and attribute inference relative to the trained mechanism's output distribution, and does not by itself protect against other classes of concerns such as fairness violations unrelated to privacy, model theft, or adversarial manipulation of training data (data poisoning), meaning DP should be understood as one component of a broader responsible machine learning practice rather than a complete privacy or safety solution on its own.
## Hyperparameter Tuning
The three DP-SGD hyperparameters, clipping norm $C$, noise multiplier $\sigma$, and batch size, interact strongly and should generally be tuned jointly rather than independently: for a fixed target privacy budget and number of training steps, increasing batch size allows a correspondingly larger noise multiplier to be used for the same effective per-step noise-to-signal ratio (since the noise is added once per batch but averaged over more examples), often improving both privacy amplification and gradient signal quality simultaneously, which is why DP training recipes frequently favor unusually large batch sizes relative to non-private training of the same model. The number of training epochs (or equivalently, total training steps) trades off directly against the cumulative privacy cost under any composition theorem, meaning DP training runs typically cannot simply be trained "until convergence" the way non-private runs often are; the number of steps must instead be fixed in advance, or bounded by early stopping against a privacy-aware budget, as part of the overall privacy accounting.
Learning rate tuning for DP-SGD often requires somewhat different values than the equivalent non-private training recipe, since the added Gaussian noise interacts with the effective step size, and published DP training recipes frequently report using learning rate schedules and warmup strategies specifically tuned for the noisy gradient regime rather than directly reusing non-private hyperparameters. Practitioners should treat the target privacy budget $\epsilon$ itself as a hyperparameter subject to a documented policy decision made with input from privacy, legal, and domain stakeholders, rather than as a purely technical tuning choice optimized solely for validation accuracy, since the choice of $\epsilon$ encodes a genuine privacy-utility tradeoff with consequences beyond model performance metrics alone.
## Real-World Applications & Case Studies
Apple has deployed local differential privacy in iOS and macOS telemetry collection since 2016, using randomized response and related local DP mechanisms to gather aggregate statistics on emoji usage, health data trends, and typing patterns across its user base without any individual device's raw data ever leaving the device in unperturbed form, representing one of the largest-scale production deployments of differential privacy to date. Google has similarly deployed differential privacy across several products, including RAPPOR for Chrome telemetry and differentially private federated learning for on-device keyboard prediction models (Gboard), combining differential privacy with federated learning's data-locality benefits to train shared models without centralizing raw user text.
The U.S. Census Bureau's adoption of differential privacy for the 2020 decennial census, replacing earlier ad hoc disclosure-avoidance techniques, represented a landmark real-world application of DP at population-statistics scale, and generated substantial public debate and research attention regarding the practical accuracy tradeoffs DP-based disclosure avoidance imposes on downstream statistical and redistricting analyses that rely on census data, illustrating both DP's practical viability at scale and the genuine, non-trivial policy tradeoffs its adoption entails. In the machine learning research community specifically, DP-SGD has been used to train differentially private variants of large language models and image classifiers, with organizations increasingly publishing DP-trained model variants alongside standard non-private versions specifically for applications involving sensitive training data such as clinical text or private user-generated content.
## Integration with Other Methods
Differential privacy and federated learning are frequently deployed together, since federated learning's core property, training without centralizing raw data, addresses data-locality and direct-access privacy concerns but does not by itself provide any formal guarantee against information leakage through the shared model updates themselves, a gap that applying differential privacy (typically DP-SGD, applied either at the per-client or per-example level) to the federated aggregation step directly addresses, combining the two techniques' complementary privacy properties into a single pipeline. Differential privacy interacts closely with knowledge distillation through frameworks such as PATE (Private Aggregation of Teacher Ensembles), which trains multiple "teacher" models on disjoint partitions of sensitive data, then uses a differentially private noisy-voting aggregation of the teachers' predictions to label public, non-sensitive data used to train a "student" model, achieving strong empirical privacy-utility tradeoffs by concentrating the DP noise addition at the teacher-aggregation step rather than throughout an entire end-to-end training process.
Differential privacy is also increasingly paired with secure multi-party computation and homomorphic encryption in systems requiring both computational privacy (preventing any party from seeing others' raw data during a joint computation) and output privacy (ensuring the final released result does not itself leak information about individual inputs), since these cryptographic techniques address a different threat model, protecting data during computation, than differential privacy's guarantee about the computation's final output, making the combination strictly more protective than either technique alone for applications with both concerns. Parameter-efficient fine-tuning methods, such as low-rank adapters, have shown particular promise when combined with DP-SGD, since applying differential privacy only to a small number of trainable adapter parameters rather than an entire large pretrained model's weights substantially improves the achievable privacy-utility tradeoff relative to full-model DP fine-tuning.
## Future Research Directions
Narrowing the privacy-utility gap for DP training of very large models remains a central open problem, with active research directions including better adaptive clipping strategies that avoid a single fixed clipping norm across all layers and training phases, improved noise-reduction techniques that exploit gradient structure and correlation across training steps, and architecture search specifically targeting DP-friendly model designs that tolerate per-example gradient noise more gracefully than standard architectures optimized purely for non-private accuracy. Better empirical auditing techniques for verifying that deployed DP mechanisms achieve their claimed formal guarantees in practice, closing potential gaps between theoretical worst-case analysis and actual implementation behavior (which has historically included real, discovered bugs in composition accounting and implementation details across multiple production DP systems), remain an active and practically important research area.
Extending rigorous differential privacy guarantees to increasingly complex, multi-stage machine learning pipelines, including retrieval-augmented generation systems, multi-agent systems, and continually updated or online-learning models where the traditional fixed-dataset, fixed-training-run framing of DP-SGD does not directly apply, is an important direction as deployed machine learning systems grow more complex and less amenable to the comparatively clean single-training-run analysis that most current DP-SGD theory and tooling assumes. Finally, developing clearer, more standardized guidance connecting formal privacy parameters to concrete, empirically measurable and communicable risk levels, bridging the persistent gap between the abstract mathematical guarantee $(\epsilon, \delta)$-DP provides and the practical, non-expert-interpretable question of "how protected is my data," remains an important interdisciplinary research direction spanning technical, legal, and policy considerations.
## Summary & Key Takeaways
Differential privacy provides a mathematically rigorous, composable guarantee bounding how much any single individual's data can influence a computation's output, formalized through the $(\epsilon, \delta)$-DP definition and achieved in practice through calibrated noise addition mechanisms such as the Laplace and Gaussian mechanisms, with smaller $\epsilon$ corresponding to stronger privacy at the cost of reduced utility. DP-SGD applies this framework to deep learning training by clipping each example's per-example gradient contribution and adding calibrated Gaussian noise to each minibatch update, with the Renyi differential privacy framework and privacy amplification by subsampling providing the tight composition analysis necessary to track cumulative privacy loss across the many thousands of steps a typical training run requires. The privacy-utility tradeoff, and its disproportionate effect on model accuracy for minority subgroups, remains DP machine learning's central practical challenge, motivating techniques such as parameter-efficient DP fine-tuning and PATE-style teacher-student distillation that concentrate privacy cost where it is most efficiently spent, and DP's real-world deployment by Apple, Google, and the U.S. Census Bureau demonstrates both its practical viability at scale and the genuine tradeoffs organizations must navigate in choosing an appropriate privacy budget.
Keywords: differential privacy, DP-SGD, epsilon-delta privacy, privacy budget, gradient clipping, Gaussian mechanism, Laplace mechanism, membership inference, Renyi differential privacy, moments accountant, privacy amplification, PATE, local differential privacy, privacy-utility tradeoff, composition theorems
---
## Appendix: Practical Labs
### Lab 1: The Laplace Mechanism Empirically Satisfies Its Epsilon-DP Bound
This lab applies the Laplace mechanism to a counting query on two neighboring datasets (differing by one record) and empirically verifies, via the ratio of sampled output densities, that the mechanism's output distributions on the two datasets remain within the theoretical $e^{\epsilon}$ bound, and that decreasing epsilon (adding more noise) tightens this ratio further.
import numpy as np
def test_laplace_mechanism_satisfies_epsilon_dp_bound():
rng = np.random.default_rng(0)
epsilon = 1.0
sensitivity = 1.0
scale = sensitivity / epsilon
true_count_D = 500
true_count_Dprime = 501
n_samples = 3_000_000
outputs_D = true_count_D + rng.laplace(loc=0.0, scale=scale, size=n_samples)
outputs_Dprime = true_count_Dprime + rng.laplace(loc=0.0, scale=scale, size=n_samples)
bins = np.linspace(495, 506, 45)
hist_D, edges = np.histogram(outputs_D, bins=bins, density=True)
hist_Dprime, _ = np.histogram(outputs_Dprime, bins=bins, density=True)
mask = (hist_D > 5e-3) & (hist_Dprime > 5e-3)
ratio = hist_D[mask] / hist_Dprime[mask]
max_ratio = ratio.max()
min_ratio = ratio.min()
print(f"epsilon={epsilon}, exp(epsilon)={np.exp(epsilon):.4f}")
print(f"empirical density ratio range: [{min_ratio:.4f}, {max_ratio:.4f}]")
tolerance = 1.2
assert max_ratio < np.exp(epsilon) * tolerance
assert min_ratio > np.exp(-epsilon) / tolerance
epsilon_small = 0.2
scale_small = sensitivity / epsilon_small
outputs_D_small = true_count_D + rng.laplace(loc=0.0, scale=scale_small, size=n_samples)
outputs_Dprime_small = true_count_Dprime + rng.laplace(loc=0.0, scale=scale_small, size=n_samples)
bins_small = np.linspace(470, 531, 62)
hist_D_small, _ = np.histogram(outputs_D_small, bins=bins_small, density=True)
hist_Dprime_small, _ = np.histogram(outputs_Dprime_small, bins=bins_small, density=True)
mask_small = (hist_D_small > 5e-4) & (hist_Dprime_small > 5e-4)
ratio_small = hist_D_small[mask_small] / hist_Dprime_small[mask_small]
print(f"smaller epsilon={epsilon_small}: density ratio range [{ratio_small.min():.4f}, {ratio_small.max():.4f}]")
assert (ratio_small.max() - 1) < (max_ratio - 1)
print("Laplace mechanism epsilon-DP bound test passed.")
if __name__ == "__main__":
test_laplace_mechanism_satisfies_epsilon_dp_bound()### Lab 2: DP-SGD Gradient Clipping Bounds Per-Example Influence, With a Privacy-Utility Tradeoff
This lab implements per-example gradient clipping and noise injection for logistic regression from scratch, verifies that clipping strictly bounds each example's gradient norm contribution, and demonstrates the expected privacy-utility tradeoff: more noise (stronger privacy) yields lower test accuracy.
import numpy as np
def test_dpsgd_clipping_bounds_per_example_gradient_contribution():
rng = np.random.default_rng(1)
n, d = 400, 6
true_w = rng.normal(size=d)
X = rng.normal(size=(n, d))
probs = 1 / (1 + np.exp(-(X @ true_w)))
y = (rng.uniform(size=n) < probs).astype(float)
def sigmoid(z):
return 1 / (1 + np.exp(-z))
def per_example_grad(w, x, yi):
p = sigmoid(x @ w)
return (p - yi) * x
def clip_grad(g, C):
norm = np.linalg.norm(g)
return g * (C / norm) if norm > C else g
def train_logreg(X, y, epochs, lr, clip_C=None, noise_multiplier=0.0, rng=None):
w = np.zeros(X.shape[1])
n = len(y)
for epoch in range(epochs):
idx = rng.permutation(n)
for i in idx:
g = per_example_grad(w, X[i], y[i])
if clip_C is not None:
g = clip_grad(g, clip_C)
if noise_multiplier > 0:
g = g + rng.normal(scale=noise_multiplier * clip_C, size=g.shape)
w = w - lr * g
return w
C = 1.0
raw_norms, clipped_norms = [], []
w0 = rng.normal(size=d) * 2
for i in range(n):
g = per_example_grad(w0, X[i], y[i])
raw_norms.append(np.linalg.norm(g))
clipped_norms.append(np.linalg.norm(clip_grad(g, C)))
raw_norms = np.array(raw_norms)
clipped_norms = np.array(clipped_norms)
print(f"raw per-example grad norm: max={raw_norms.max():.3f} mean={raw_norms.mean():.3f}")
print(f"clipped per-example grad norm: max={clipped_norms.max():.3f}")
assert clipped_norms.max() <= C * 1.0001
assert raw_norms.max() > C
X_test = rng.normal(size=(300, d))
y_test = (rng.uniform(size=300) < sigmoid(X_test @ true_w)).astype(float)
def test_accuracy(w):
preds = (sigmoid(X_test @ w) > 0.5).astype(float)
return np.mean(preds == y_test)
w_no_dp = train_logreg(X, y, epochs=8, lr=0.1, clip_C=None, noise_multiplier=0.0,
rng=np.random.default_rng(10))
w_low_noise = train_logreg(X, y, epochs=8, lr=0.1, clip_C=C, noise_multiplier=0.3,
rng=np.random.default_rng(10))
w_high_noise = train_logreg(X, y, epochs=8, lr=0.1, clip_C=C, noise_multiplier=3.0,
rng=np.random.default_rng(10))
acc_no_dp = test_accuracy(w_no_dp)
acc_low_noise = test_accuracy(w_low_noise)
acc_high_noise = test_accuracy(w_high_noise)
print(f"test accuracy: no-DP={acc_no_dp:.3f} low-noise-DP={acc_low_noise:.3f} high-noise-DP={acc_high_noise:.3f}")
assert acc_no_dp >= acc_low_noise - 0.05
assert acc_low_noise > acc_high_noise
print("DP-SGD clipping and privacy-utility tradeoff test passed.")
if __name__ == "__main__":
test_dpsgd_clipping_bounds_per_example_gradient_contribution()### Lab 3: Membership Inference Attacks Succeed More Against Overfit Models
This lab trains an overfit logistic regression model (small training set, no regularization, many epochs) and a DP-style regularized-and-noised counterpart, then runs a confidence-thresholding membership inference attack against both, verifying the attack achieves higher accuracy against the overfit model, consistent with its larger train-test generalization gap.
import numpy as np
def test_membership_inference_attack_succeeds_more_against_overfit_model():
rng = np.random.default_rng(2)
d = 25
n_train = 50
n_holdout = 500
true_w = rng.normal(size=d) * 0.6
def sigmoid(z):
return 1 / (1 + np.exp(-z))
def gen_data(n):
X = rng.normal(size=(n, d))
y = (rng.uniform(size=n) < sigmoid(X @ true_w)).astype(float)
return X, y
X_train, y_train = gen_data(n_train)
X_holdout, y_holdout = gen_data(n_holdout)
def train_logreg(X, y, epochs, lr, l2=0.0, noise_std=0.0, rng=None):
n, d = X.shape
w = np.zeros(d)
for epoch in range(epochs):
p = sigmoid(X @ w)
grad = X.T @ (p - y) / n + l2 * w
if noise_std > 0:
grad = grad + rng.normal(scale=noise_std, size=grad.shape)
w = w - lr * grad
return w
w_overfit = train_logreg(X_train, y_train, epochs=1000, lr=0.5, l2=0.0)
w_dp = train_logreg(X_train, y_train, epochs=200, lr=0.3, l2=0.5, noise_std=0.05,
rng=np.random.default_rng(20))
def per_example_loss(w, X, y):
p = sigmoid(X @ w)
p = np.clip(p, 1e-8, 1 - 1e-8)
return -(y * np.log(p) + (1 - y) * np.log(1 - p))
def attack_balanced_accuracy(w, X_mem, y_mem, X_nonmem, y_nonmem, threshold):
loss_mem = per_example_loss(w, X_mem, y_mem)
loss_nonmem = per_example_loss(w, X_nonmem, y_nonmem)
tpr = np.mean(loss_mem < threshold)
tnr = np.mean(loss_nonmem >= threshold)
return 0.5 * (tpr + tnr)
X_nonmem = X_holdout[:250]
y_nonmem = y_holdout[:250]
def best_attack_accuracy(w):
loss_mem = per_example_loss(w, X_train, y_train)
candidate_thresholds = np.quantile(loss_mem, np.linspace(0.05, 0.95, 30))
best = 0.0
for t in candidate_thresholds:
acc = attack_balanced_accuracy(w, X_train, y_train, X_nonmem, y_nonmem, t)
best = max(best, acc)
return best
acc_overfit = best_attack_accuracy(w_overfit)
acc_dp = best_attack_accuracy(w_dp)
train_loss_overfit = per_example_loss(w_overfit, X_train, y_train).mean()
holdout_loss_overfit = per_example_loss(w_overfit, X_holdout, y_holdout).mean()
train_loss_dp = per_example_loss(w_dp, X_train, y_train).mean()
holdout_loss_dp = per_example_loss(w_dp, X_holdout, y_holdout).mean()
print(f"overfit model: train_loss={train_loss_overfit:.4f} holdout_loss={holdout_loss_overfit:.4f} gap={holdout_loss_overfit - train_loss_overfit:.4f}")
print(f"dp-style model: train_loss={train_loss_dp:.4f} holdout_loss={holdout_loss_dp:.4f} gap={holdout_loss_dp - train_loss_dp:.4f}")
print(f"best-threshold membership inference attack accuracy: overfit={acc_overfit:.4f} dp-style={acc_dp:.4f}")
assert (holdout_loss_overfit - train_loss_overfit) > (holdout_loss_dp - train_loss_dp)
assert acc_overfit > acc_dp
assert acc_overfit > 0.55
print("Membership inference attack test passed.")
if __name__ == "__main__":
test_membership_inference_attack_succeeds_more_against_overfit_model()### Lab 4: Advanced Composition Gives a Tighter Privacy Budget Than Basic Composition at Scale
This lab computes the total privacy cost of many sequentially composed Gaussian-mechanism-style applications under both basic composition (linear in the number of compositions) and advanced composition (closer to square-root scaling), verifying that advanced composition's relative advantage grows as the number of compositions increases, which is what makes training deep models over many thousands of DP-SGD steps practically feasible.
import numpy as np
def test_advanced_composition_gives_tighter_privacy_budget_for_many_compositions():
eps_0 = 0.05
delta_prime = 1e-5
k_values = [10, 100, 1000, 10000]
basic_totals = []
advanced_totals = []
for k in k_values:
basic_eps = k * eps_0
advanced_eps = (
np.sqrt(2 * k * np.log(1 / delta_prime)) * eps_0
+ k * eps_0 * (np.exp(eps_0) - 1)
)
basic_totals.append(basic_eps)
advanced_totals.append(advanced_eps)
print(f"k={k:6d}: basic composition eps={basic_eps:10.3f} advanced composition eps={advanced_eps:10.3f}")
basic_totals = np.array(basic_totals)
advanced_totals = np.array(advanced_totals)
ratio = advanced_totals / basic_totals
print(f"advanced/basic ratio at each k: {ratio}")
assert advanced_totals[-1] < basic_totals[-1]
assert advanced_totals[-2] < basic_totals[-2]
assert np.all(np.diff(ratio) < 0)
assert ratio[-1] < 0.15
print("Advanced composition tighter-bound-at-scale test passed.")
if __name__ == "__main__":
test_advanced_composition_gives_tighter_privacy_budget_for_many_compositions()