Federated Learning Privacy
# Federated Learning & Privacy
## Introduction & Motivation
Federated Learning: train models across decentralized data without sharing raw data. Privacy-preserving ML. Applications: healthcare, finance, mobile devices.
Motivation: Protect privacy; train on distributed data.
Applications: Healthcare records, financial data, mobile networks.
---
## Core Concepts & Theory
### Federated Averaging (FedAvg)
Aggregate local model updates.
### Differential Privacy
Add noise to hide individual records.
### Secure Aggregation
Cryptographic protection during aggregation.
### Communication Efficiency
Reduce bandwidth for model updates.
---
## Mathematical Formulation
Federated Averaging:
$$w^{t+1} = \frac{1}{K} \sum_{k=1}^{K} w_k^{t+1}$$
Differential Privacy (DP):
$$\Delta w = \frac{
abla L + \xi}{\max(1, \frac{\|
abla L\|_2}{C})}$$
Privacy Budget (ε, δ):
$$P(A(D)) \approx e^\epsilon P(A(D'))$$
---
## Advanced Theory & Extensions
### FedProx
Proximal term for heterogeneous data.
### FedSGD
Stochastic gradient descent variant.
### Secure Aggregation
Cryptographic protocols.
---
## Computational Considerations
Communication: O(num_rounds·model_size).
Local computation: O(local_data·iterations).
Aggregation: O(num_clients·model_size).
---
## Practical Implementation Strategies
### Client Selection
Random or stratified sampling.
### Local Epochs
Multiple passes on local data.
### Differential Privacy
Add noise for privacy-accuracy trade-off.
---
## Benchmark Datasets & Evaluation
FEMNIST: Federated digit recognition.
Shakespeare: Federated language modeling.
Synthetic Federated Data: Controlled heterogeneity.
---
## Key Challenges & Limitations
### Data Heterogeneity
Non-IID data across clients.
### Communication Cost
Limited bandwidth.
### Privacy-Accuracy Trade-off
Noise reduces utility.
---
## Hyperparameter Tuning
Noise scale (σ): 0.01-0.1.
Privacy budget (ε): 1-10.
Aggregation frequency: 1-100 rounds.
---
## Real-World Applications & Case Studies
Google Keyboard: Federated learning on devices.
Healthcare: Hospital network collaboration.
Finance: Multi-institution data sharing.
---
## Integration with Other Methods
Federated learning + differential privacy for strong guarantees; + compression for efficiency.
---
## Summary & Key Takeaways
Federated Learning via distributed averaging and differential privacy enables privacy-preserving collaborative training.
Principles:
1. Decentralized data: No centralization.
2. Federated averaging: Local update aggregation.
3. Differential privacy: Statistical privacy.
4. Communication efficiency: Bandwidth reduction.
5. Heterogeneity: Non-IID data handling.
---
---
## Appendix: Practical Labs
### Lab 1: Federated Averaging
import numpy as np
def federated_averaging(client_models, weights=None):
"""Average weights from multiple clients"""
if weights is None:
weights = [1/len(client_models)] * len(client_models)
avg_model = None
for i, model_weights in enumerate(client_models):
if avg_model is None:
avg_model = model_weights.copy() * weights[i]
else:
avg_model += model_weights * weights[i]
return avg_model
# Test
np.random.seed(42)
models = [np.random.randn(50), np.random.randn(50), np.random.randn(50)]
avg = federated_averaging(models)
assert avg.shape == (50,), "Average shape"
print("✓ Federated averaging working")
if __name__ == "__main__":
print("Lab 1: FederatedAveraging - PASSED")### Lab 2: Differential Privacy Noise
import numpy as np
def add_differential_privacy(gradients, sensitivity, epsilon, delta):
"""Add Laplace noise for differential privacy"""
# Laplace noise scale
scale = sensitivity / epsilon
# Add noise
noise = np.random.laplace(0, scale, size=gradients.shape)
noisy_gradients = gradients + noise
return noisy_gradients
# Test
np.random.seed(42)
grads = np.random.randn(50)
noisy = add_differential_privacy(grads, sensitivity=1.0, epsilon=1.0, delta=1e-6)
assert noisy.shape == grads.shape, "Shape preserved"
assert not np.allclose(noisy, grads), "Noise added"
print("✓ Differential privacy working")
if __name__ == "__main__":
print("Lab 2: DifferentialPrivacy - PASSED")### Lab 3: Local Training
import numpy as np
def federated_local_update(local_data, initial_weights, num_epochs=5, lr=0.01):
"""Perform local training on client data"""
weights = initial_weights.copy()
for epoch in range(num_epochs):
# Simulate gradient computation
gradient = -local_data @ weights + np.random.randn(weights.shape[0]) * 0.01
# Update
weights -= lr * gradient
return weights
# Test
np.random.seed(42)
local_data = np.random.randn(100, 50)
init_weights = np.random.randn(50)
updated = federated_local_update(local_data, init_weights, num_epochs=3)
assert updated.shape == init_weights.shape, "Shape preserved"
print("✓ Local training working")
if __name__ == "__main__":
print("Lab 3: LocalUpdate - PASSED")### Lab 4: Communication Compression
import numpy as np
def compress_gradients(gradients, compression_ratio=0.1):
"""Compress gradients by keeping top-k"""
k = max(1, int(gradients.size * compression_ratio))
# Keep top-k by magnitude
flat = gradients.flatten()
indices = np.argsort(np.abs(flat))[-k:]
# Create sparse representation
compressed = np.zeros_like(flat)
compressed[indices] = flat[indices]
return compressed.reshape(gradients.shape)
# Test
np.random.seed(42)
grads = np.random.randn(50, 50)
compressed = compress_gradients(grads, compression_ratio=0.1)
sparsity = 1 - (np.count_nonzero(compressed) / compressed.size)
assert sparsity > 0.8, "Compression achieved"
print("✓ Gradient compression working")
if __name__ == "__main__":
print("Lab 4: GradientCompression - PASSED")