local sgd
**Local SGD** is a distributed training algorithm that **performs multiple gradient updates locally before synchronizing** — dramatically reducing communication overhead in distributed and federated learning by allowing workers to train independently for H steps before averaging parameters, making distributed training practical over slow networks.
**What Is Local SGD?**
- **Definition**: Distributed optimization with periodic synchronization.
- **Algorithm**: Each worker performs H local SGD steps, then synchronizes.
- **Goal**: Reduce communication rounds by H× while maintaining convergence.
- **Also Known As**: FedAvg (Federated Averaging) in federated learning context.
**Why Local SGD Matters**
- **Communication Efficiency**: H× reduction in communication rounds.
- **Slow Network Tolerance**: Works with commodity networks, not just high-speed interconnects.
- **Straggler Handling**: Slow workers don't block others during local phase.
- **Federated Learning Enabler**: Makes training on mobile devices practical.
- **Cost Reduction**: Less communication = lower cloud egress costs.
**Algorithm**
**Initialization**:
- All workers start with same model parameters θ_0.
- Agree on local steps H and learning rate schedule.
**Training Loop**:
```
For round t = 1, 2, 3, ...:
// Local training phase
Each worker k independently:
For h = 1 to H:
Sample mini-batch from local data
Compute gradient g_k
Update: θ_k ← θ_k - η · g_k
// Synchronization phase
Aggregate: θ_global ← (1/K) Σ_k θ_k
Broadcast θ_global to all workers
```
**Key Parameters**:
- **H (local steps)**: Number of SGD steps between synchronizations.
- **K (workers)**: Number of parallel workers.
- **η (learning rate)**: Step size for local updates.
**Convergence Analysis**
**Convergence Guarantee**:
- Converges to same solution as standard SGD (under assumptions).
- Convergence rate: O(1/√(KHT)) for convex, O(1/√(KHT)) for non-convex.
- Requires learning rate adjustment for large H.
**Key Insights**:
- **Worker Divergence**: Local models diverge during local phase.
- **Synchronization Corrects**: Averaging brings models back together.
- **Trade-Off**: Larger H → more divergence but less communication.
**Optimal H Selection**:
- Too small: Excessive communication overhead.
- Too large: Worker divergence hurts convergence.
- Typical: H = 10-100 for datacenter, H = 100-1000 for federated.
**Comparison with Other Methods**
**vs. Synchronous SGD**:
- **Local SGD**: H local steps, then sync (H=1 is sync SGD).
- **Sync SGD**: Every step synchronized.
- **Trade-Off**: Local SGD reduces communication, slightly slower convergence.
**vs. Asynchronous SGD**:
- **Local SGD**: Periodic synchronization, bounded staleness.
- **Async SGD**: Continuous asynchronous updates, unbounded staleness.
- **Trade-Off**: Local SGD more stable, async SGD more communication efficient.
**vs. Gradient Compression**:
- **Local SGD**: Reduce communication frequency.
- **Compression**: Reduce communication size per round.
- **Combination**: Can use both together for maximum efficiency.
**Variants & Extensions**
**Adaptive H Selection**:
- Dynamically adjust H based on worker divergence.
- Increase H when models are similar, decrease when diverging.
- Improves convergence while maintaining communication efficiency.
**Periodic Averaging Schedules**:
- Exponentially increasing H: H = 1, 2, 4, 8, ...
- Allows frequent sync early, less frequent later.
- Balances exploration and communication.
**Momentum-Based Local SGD**:
- Add momentum to local updates.
- Helps overcome local minima during local phase.
- Improves convergence quality.
**Applications**
**Datacenter Distributed Training**:
- Train large models across GPU clusters.
- Reduce network bottleneck in multi-node training.
- Typical: H = 10-50 for fast interconnects.
**Federated Learning**:
- Train on mobile devices with slow, intermittent connections.
- FedAvg is essentially Local SGD for federated setting.
- Typical: H = 100-1000 for mobile devices.
**Edge Computing**:
- Train on edge devices with limited connectivity.
- Periodic synchronization with cloud server.
- Balances local computation and communication.
**Practical Considerations**
**Learning Rate Tuning**:
- Larger H may require learning rate adjustment.
- Rule of thumb: Scale learning rate by √H or keep constant.
- Warmup helps stabilize early training.
**Batch Size**:
- Local batch size affects convergence.
- Larger local batches can compensate for larger H.
- Trade-off: Memory vs. convergence speed.
**Non-IID Data**:
- Worker data distributions may differ (federated learning).
- Non-IID data increases worker divergence.
- May need smaller H or additional regularization.
**Tools & Implementations**
- **PyTorch Distributed**: Easy implementation with DDP.
- **TensorFlow Federated**: Built-in FedAvg (Local SGD).
- **Horovod**: Supports periodic averaging for Local SGD.
- **Custom**: Simple to implement with any distributed framework.
**Best Practices**
- **Start with H=1**: Verify convergence, then increase H.
- **Monitor Divergence**: Track worker model differences.
- **Tune Learning Rate**: Adjust for your specific H value.
- **Use Warmup**: Stabilize early training with frequent sync.
- **Combine with Compression**: Maximize communication efficiency.
Local SGD is **the foundation of practical distributed training** — by allowing workers to train independently between synchronizations, it makes distributed learning feasible over slow networks and enables federated learning on mobile devices, transforming how we train large-scale machine learning models.