Differential Privacy in Machine Learning

# Differential Privacy in Machine Learning

## Introduction & Motivation

Differential privacy: provide mathematical privacy guarantees. Protect individual data while learning. Applications: privacy-preserving models, federated learning.

Motivation: Enable training while protecting individual privacy.

Applications: Healthcare ML, financial modeling, user data protection.

---

## Core Concepts & Theory

### Privacy Budget

Quantify privacy loss.

### Noise Addition

Protect individual samples.

### DP-SGD

Private gradient descent.

### Utility-Privacy Tradeoff

Balance accuracy and privacy.

---

## Mathematical Formulation

Differential Privacy:
$$P(M(D) \in S) \leq e^\epsilon P(M(D') \in S) + \delta$$

DP-SGD Noise:
$$ heta_t = heta_{t-1} - \eta( abla L + N(0, \sigma^2))$$

Privacy Accounting:
$$\epsilon = \sum_t \frac{q_t}{\sigma_t}$$

---

## Advanced Theory & Extensions

### Gradient Clipping

Bound gradient norms.

### Moment Accountant

Tighter privacy bounds.

### Federated DP

Distributed privacy.

---

## Computational Considerations

Noise overhead: ~5-10% accuracy loss.

Computation: O(batch size).

Privacy budget: Decreases with gradient updates.

---

## Practical Implementation Strategies

### Noise Schedule

Vary noise over time.

### Privacy Budgeting

Track epsilon accumulation.

### Gradient Clipping

Clip large gradients.

---

## Benchmark Datasets & Evaluation

MNIST: Privacy-preserving classification.

CIFAR-10: Large-scale private learning.

Custom benchmarks: Privacy-utility curves.

---

## Key Challenges & Limitations

### Accuracy Loss

Privacy hurts performance.

### Privacy Budget

Finite total privacy.

### Convergence

Slower with noise.

---

## Hyperparameter Tuning

Noise scale: 0.1-1.0.

Gradient clip: 1.0-5.0.

Privacy budget: Task-dependent.

---

## Real-World Applications & Case Studies

Healthcare: Protect patient data.

Finance: Secure financial modeling.

Recommendation: Private personalization.

---

## Integration with Other Methods

Differential privacy + federated learning; + secure computation.

---

## Summary & Key Takeaways

Differential privacy enables privacy-preserving learning.

Principles:
1. Privacy guarantee: Mathematical bounds.
2. Noise addition: Protect individuals.
3. DP-SGD: Private optimization.
4. Privacy budget: Track accumulation.
5. Tradeoff: Accuracy vs privacy.

---

## Appendix: Practical Labs

### Lab 1: Add Gaussian Noise

import numpy as np

def add_dp_noise(gradient, noise_scale=0.1):
 """Add noise for differential privacy"""
 noise = np.random.normal(0, noise_scale, gradient.shape)
 noisy_grad = gradient + noise
 return noisy_grad

np.random.seed(42)
grad = np.random.randn(100)
noisy = add_dp_noise(grad, 0.1)
assert noisy.shape == grad.shape
assert not np.allclose(grad, noisy)
print("✓ DP noise addition working")

### Lab 2: Gradient Clipping

import numpy as np

def clip_gradients(gradients, clip_norm=1.0):
 """Clip gradients by norm"""
 total_norm = np.linalg.norm(gradients)
 if total_norm > clip_norm:
 gradients = gradients * (clip_norm / total_norm)
 return gradients

np.random.seed(42)
grad = np.random.randn(1000)
clipped = clip_gradients(grad, 1.0)
assert np.linalg.norm(clipped) <= 1.0
print("✓ Gradient clipping working")

### Lab 3: Privacy Accounting

import numpy as np

def compute_privacy(batch_size, num_steps, noise_scale, dataset_size):
 """Compute privacy budget"""
 sampling_prob = batch_size / dataset_size
 rdp = []
 
 for order in range(2, 33):
 coeff = (sampling_prob ** order) / (2 * (noise_scale ** 2))
 rdp.append(coeff * num_steps)
 
 # Convert RDP to epsilon
 epsilon = min(rdp) / (max(range(2, 33)) - 1)
 return epsilon

eps = compute_privacy(32, 1000, 1.0, 60000)
assert eps > 0
print(f"✓ Privacy budget: ε={eps:.3f}")

### Lab 4: Utility-Privacy Curve

import numpy as np

def privacy_utility_tradeoff(noise_scales):
 """Compute accuracy vs privacy tradeoff"""
 accuracies = []
 privacies = []
 
 for sigma in noise_scales:
 # Simulated accuracy decrease with noise
 acc = 0.95 - 0.1 * sigma
 eps = 1.0 / sigma
 
 accuracies.append(acc)
 privacies.append(eps)
 
 return accuracies, privacies

scales = np.linspace(0.1, 1.0, 5)
acc, priv = privacy_utility_tradeoff(scales)
assert len(acc) == len(priv)
print("✓ Privacy-utility tradeoff computed")

---