Federated Learning is the distributed machine learning paradigm where multiple clients (mobile devices, hospitals, organizations) collaboratively train a shared model without sharing their raw data — each client trains on local data and sends only model updates (gradients or weights) to a central server that aggregates them, preserving data privacy and data sovereignty while enabling model training across decentralized datasets that cannot be centralized due to privacy regulations (GDPR, HIPAA), competitive concerns, or communication constraints.

Federated Averaging (FedAvg)

The foundational algorithm (McMahan et al., Google, 2017): 1. Server broadcasts current global model W_t to a subset of clients (10-1000 per round). 2. Each selected client trains the model on its local data for E local epochs (E=1-5) using SGD. 3. Each client sends its updated model W_t^k back to the server. 4. Server aggregates: W_{t+1} = Σ_k (n_k/n) × W_t^k (weighted average by dataset size). 5. Repeat for 100-1000 communication rounds.

Communication efficiency: instead of sending gradient updates every batch (100K batches per epoch), each client sends one model update per round after E full epochs — 1000-100,000× fewer messages.

Challenges

Non-IID Data: Different clients have different data distributions. A hospital in Japan has different patient demographics than one in Nigeria. Non-IID data causes client models to diverge — averaging divergent models can produce a worse global model than any individual client's model.

• Solutions: FedProx (add proximal term penalizing divergence from global model), SCAFFOLD (variance reduction using control variates), personalization layers (shared backbone + client-specific heads).

Communication Efficiency: Model updates are large (hundreds of MB for modern models). Mobile networks have limited bandwidth.

• Solutions: Gradient compression (top-K sparsification: send only the largest 1-10% of gradients), quantization (send INT8 instead of FP32 gradients), knowledge distillation (send predictions instead of model updates).

Privacy Guarantees

FedAvg alone does not guarantee privacy — model updates can leak information:

• Gradient Inversion Attacks: Given model gradients, reconstruct training images with high fidelity. Particularly effective for small batch sizes.
• Secure Aggregation: Cryptographic protocol where the server sees only the sum of client updates, not individual updates. Uses secret sharing or homomorphic encryption.
• Differential Privacy (DP-FedAvg): Clip each client's update to bounded norm, add calibrated Gaussian noise. Provides (ε, δ)-differential privacy — mathematically bounded information leakage. Trade-off: noise reduces model accuracy (typically 1-3% on vision tasks with ε=8).

Applications

• Google Gboard: Next-word prediction model trained on millions of Android devices without collecting keystroke data. The canonical federated learning deployment.
• Healthcare: Multi-hospital model training (FeTS for brain tumor segmentation across 71 institutions worldwide). Each hospital keeps patient data on-premises. Model quality approaches centralized training.
• Financial: Cross-institution fraud detection without sharing transaction data between competing banks.

Federated Learning is the privacy-preserving paradigm that enables collaborative AI without data centralization — the technical infrastructure for training models across organizational and regulatory boundaries, proving that strong AI and strong privacy are not mutually exclusive.