Federated Learning (FL) is the distributed machine learning paradigm where models are trained across multiple decentralized devices or institutions without centralizing the raw data — each participant trains locally on their private data and shares only model updates (gradients or weights) with a central server that aggregates them, preserving data privacy while enabling collaborative model improvement across organizational and regulatory boundaries.

Why Federated Learning Exists

Traditional ML requires centralizing all training data in one location. This is impossible when:

• Regulatory constraints: GDPR, HIPAA, or CCPA prohibit data sharing across jurisdictions or organizations.
• Privacy sensitivity: Medical records, financial transactions, and personal communications cannot leave the source device/institution.
• Data volume: Mobile devices collectively hold petabytes of data that is impractical to centralize.
• Competitive concerns: Multiple hospitals want to collaboratively train a better diagnostic model without sharing their patients' data with competitors.

Federated Averaging (FedAvg)

The foundational FL algorithm: 1. Server sends the current global model to a random subset of clients. 2. Each client trains the model on its local data for E epochs (local SGD). 3. Clients send their updated model weights (or weight deltas) back to the server. 4. Server averages the client updates: w_global = (1/K) Σ wₖ, weighted by each client's dataset size. 5. Repeat until convergence.

Challenges and Solutions

• Non-IID Data: Client datasets have different distributions (a hospital specializing in cardiac cases vs. oncology). FedAvg can diverge. Solutions: FedProx (proximal regularization), SCAFFOLD (variance reduction), personalized federated learning (per-client adaptation layers).

• Communication Efficiency: Sending full model updates (hundreds of MB for large models) is expensive over mobile networks. Solutions: gradient compression (top-K sparsification, quantization), federated distillation (share logits instead of weights), increasing local computation (E>1) to reduce round trips.

• Client Heterogeneity: Devices have different compute capabilities and availability. Asynchronous FL allows clients to contribute updates at their own pace; knowledge distillation enables different model architectures per client.

• Privacy Attacks: Even without raw data, model gradients can leak information (gradient inversion attacks can reconstruct training images). Defenses:
• Differential Privacy: Add calibrated noise to gradient updates, providing mathematical privacy guarantees (ε-differential privacy).
• Secure Aggregation: Cryptographic protocols ensure the server can compute the aggregate without seeing individual client updates.
• Trusted Execution Environments: Hardware enclaves (Intel SGX) process aggregation in isolated, verifiable environments.

Production Deployments

• Google Gboard: Next-word prediction trained across millions of Android devices using federated learning. The model improves from global keyboard usage without Google seeing what users type.
• Apple: On-device ML models for Siri, QuickType, and photo features trained using privacy-preserving federated approaches.

Federated Learning is the privacy-preserving training paradigm that resolves the fundamental tension between data-hungry ML and data-protective regulation — enabling models to learn from the world's distributed data without that data ever leaving its source.