Federated Learning is the distributed machine learning paradigm where multiple clients (devices or organizations) collaboratively train a shared model without exchanging their raw data — each client trains locally on its own data and sends only model updates (gradients or weights) to a central server for aggregation, preserving data privacy while enabling learning from datasets that could never be centralized due to legal, competitive, or logistical constraints.

The Privacy Motivation

Traditional ML requires centralizing all training data on one server — impossible when data is medical records across hospitals (HIPAA), financial transactions across banks (GDPR), or user interactions on personal devices (privacy expectations). Federated learning keeps data where it is, training happens at the data source.

FedAvg: The Foundational Algorithm

1. Server broadcasts the current global model to a random subset of clients. 2. Each client trains the model on its local data for several epochs (local SGD). 3. Clients send updated model weights (or weight deltas) back to the server. 4. Server aggregates updates by weighted averaging (weighted by each client's dataset size): w_global = Σ(n_k/n) × w_k. 5. Repeat until convergence.

Multiple local epochs reduce communication rounds (the dominant cost), but introduce client drift — local models specialize to their local data distribution, potentially diverging from the global optimum.

Key Challenges

• Non-IID Data: Each client's data distribution may be fundamentally different (a hospital in Mumbai sees different diseases than one in Stockholm). Non-IID data causes FedAvg to converge slowly or to suboptimal solutions. Mitigation: FedProx (proximal term penalizing divergence from global model), SCAFFOLD (variance reduction), personalization layers.
• Communication Efficiency: Sending full model weights (billions of parameters for LLMs) every round is prohibitive. Techniques: gradient compression (top-K sparsification), quantization (1-bit SGD), local SGD with infrequent synchronization.
• Heterogeneous Compute: Clients range from flagship smartphones to low-end IoT devices. Stragglers slow synchronous rounds. Solutions: asynchronous aggregation, partial model training (smaller models on weaker devices).
• Privacy Guarantees: Model updates can leak information about training data (gradient inversion attacks can reconstruct images from gradients). Differential privacy (adding calibrated noise to updates) provides formal privacy guarantees at the cost of model accuracy.

Applications

• Mobile Keyboard Prediction (Google Gboard): Next-word prediction trained across millions of devices without collecting user typing data.
• Healthcare: Multi-hospital model training for medical imaging (tumor detection, drug discovery) without sharing patient records.
• Financial Fraud Detection: Banks collaboratively train fraud models without sharing transaction data.

Federated Learning is the paradigm that makes machine learning possible where data centralization is impossible — enabling collaborative model training across organizational and jurisdictional boundaries while keeping sensitive data under its owner's control.