FedAvg (Federated Averaging) is the foundational algorithm for federated learning — each client performs multiple local SGD steps on their private data, then sends the updated model (or model delta) to the server, which averages the updates to produce the new global model.

FedAvg Algorithm

• Server: Send global model $w_t$ to a random subset of clients.
• Client: Each client $k$ runs $E$ epochs of SGD on their local data: $w_k = w_t - eta \nabla L_k(w_t)$ (local training).
• Communication: Each client sends $w_k - w_t$ (model delta) to the server.
• Aggregation: Server averages: $w_{t+1} = w_t + frac{1}{K}sum_{k=1}^K (w_k - w_t)$ (weighted by dataset size).

Why It Matters

• Communication Efficient: Multiple local steps per communication round dramatically reduce communication.
• Privacy: Raw data never leaves the clients — only model updates are shared.
• Heterogeneity Challenge: Non-IID data across clients can cause FedAvg to diverge — motivating FedProx and SCAFFOLD.

FedAvg is the workhorse of federated learning — averaging locally trained models for collaborative learning without data sharing.