horizontal federated learning

**Horizontal Federated Learning** is the standard federated learning setting where **distributed clients have the same features but different samples** — enabling organizations with compatible data schemas but separate user populations to collaboratively train models while keeping data decentralized, the most common federated learning scenario in practice. **What Is Horizontal Federated Learning?** - **Definition**: Federated learning where data is partitioned by samples (users/examples). - **Feature Space**: All clients have same features/columns. - **Sample Space**: Each client has different samples/rows. - **Also Known As**: Sample-partitioned federated learning. **Why Horizontal Federated Learning Matters** - **Most Common Scenario**: Matches real-world federated deployments. - **Natural Data Distribution**: Users naturally partitioned across devices/institutions. - **Privacy Preservation**: Keep user data on local devices/servers. - **Regulatory Compliance**: Meet data residency and privacy requirements. - **Scalability**: Train on billions of devices without centralizing data. **Characteristics** **Data Distribution**: - **Same Features**: All clients measure same attributes. - **Different Samples**: Each client has different users/examples. - **Example**: Multiple hospitals with same patient measurements but different patients. **Model Architecture**: - **Shared Architecture**: All clients use identical model structure. - **Compatible Parameters**: Model parameters can be directly averaged. - **Aggregation**: Simple parameter averaging works naturally. **Contrast with Vertical FL**: - **Horizontal**: Same features, different samples (user-partitioned). - **Vertical**: Different features, overlapping samples (feature-partitioned). - **Example**: Horizontal = multiple banks with same customer data schema; Vertical = bank + retailer with shared customers. **Standard Algorithms** **FedAvg (Federated Averaging)**: - **Most Popular**: De facto standard for horizontal FL. - **Process**: Clients train locally, server averages parameters. - **Simple**: Easy to implement and understand. - **Effective**: Works well in practice despite simplicity. **FedProx**: - **Extension**: Adds proximal term to handle heterogeneity. - **Regularization**: Keeps local updates close to global model. - **Benefit**: More robust to non-IID data and stragglers. **FedOpt**: - **Server Optimization**: Apply adaptive optimizers (Adam, Yogi) at server. - **Client SGD**: Clients still use SGD locally. - **Benefit**: Faster convergence, better handling of heterogeneity. **Applications** **Mobile Devices**: - **Use Case**: Next-word prediction, voice recognition, app recommendations. - **Example**: Google Gboard keyboard training across millions of phones. - **Data**: Each phone has user's typing patterns, voice samples. - **Benefit**: Personalized models without uploading sensitive data. **Healthcare Institutions**: - **Use Case**: Disease prediction, treatment recommendations, medical imaging. - **Example**: Multiple hospitals collaborating on diagnosis models. - **Data**: Each hospital has patient records with same measurements. - **Benefit**: Larger training dataset without violating HIPAA. **Financial Organizations**: - **Use Case**: Fraud detection, credit scoring, risk assessment. - **Example**: Banks collaborating on fraud detection. - **Data**: Each bank has transaction records with same features. - **Benefit**: Better models without sharing customer data. **IoT Devices**: - **Use Case**: Predictive maintenance, anomaly detection. - **Example**: Smart home devices learning usage patterns. - **Data**: Each device has sensor readings with same schema. - **Benefit**: Collective intelligence without cloud upload. **Challenges** **Non-IID Data**: - **Problem**: Client data distributions differ significantly. - **Impact**: Slower convergence, reduced accuracy. - **Solutions**: FedProx, data augmentation, personalization. **Communication Efficiency**: - **Problem**: Frequent communication with many clients is expensive. - **Impact**: Bandwidth costs, latency, energy consumption. - **Solutions**: Local SGD, gradient compression, client sampling. **Stragglers**: - **Problem**: Slow clients delay training rounds. - **Impact**: Increased training time, resource waste. - **Solutions**: Asynchronous updates, timeout mechanisms, client selection. **Privacy & Security**: - **Problem**: Model updates may leak information about training data. - **Impact**: Privacy violations, inference attacks. - **Solutions**: Secure aggregation, differential privacy, encrypted computation. **System Heterogeneity**: - **Problem**: Clients have different computational capabilities. - **Impact**: Uneven participation, fairness issues. - **Solutions**: Adaptive model sizes, tiered participation. **Technical Components** **Client Selection**: - **Random Sampling**: Select subset of clients each round. - **Stratified Sampling**: Ensure diverse client representation. - **Importance Sampling**: Prioritize clients with more data or higher loss. **Aggregation Methods**: - **Simple Average**: θ_global = (1/K) Σ_k θ_k. - **Weighted Average**: θ_global = Σ_k (n_k/n) θ_k (weight by data size). - **Robust Aggregation**: Median, trimmed mean to handle outliers. **Privacy Mechanisms**: - **Secure Aggregation**: Cryptographic protocol hiding individual updates. - **Differential Privacy**: Add calibrated noise to updates. - **Homomorphic Encryption**: Compute on encrypted updates. **Communication Optimization**: - **Gradient Compression**: Quantization, sparsification, low-rank. - **Local Steps**: Multiple local updates before communication (Local SGD). - **Model Compression**: Distillation, pruning for smaller models. **Evaluation Metrics** **Model Performance**: - **Global Test Accuracy**: Performance on held-out centralized test set. - **Local Test Accuracy**: Average performance on client test sets. - **Fairness**: Variance in performance across clients. **Efficiency Metrics**: - **Communication Rounds**: Number of server-client communication cycles. - **Total Communication**: Bytes transferred (upload + download). - **Training Time**: Wall-clock time to convergence. **Privacy Metrics**: - **Privacy Budget**: ε in differential privacy. - **Membership Inference**: Success rate of privacy attacks. - **Reconstruction Error**: Ability to recover training data. **Tools & Frameworks** - **TensorFlow Federated**: Google's production-grade FL framework. - **PySyft**: OpenMined's privacy-preserving ML library. - **Flower**: Flexible and scalable FL framework. - **FedML**: Comprehensive research and production FL platform. - **FATE**: Industrial federated learning framework. **Best Practices** - **Start Simple**: Begin with FedAvg, add complexity as needed. - **Monitor Heterogeneity**: Track data distribution differences across clients. - **Tune Hyperparameters**: Learning rate, local steps, client sampling rate. - **Implement Privacy**: Use secure aggregation and differential privacy. - **Handle Failures**: Design for client dropouts and network issues. - **Evaluate Fairly**: Report both global and per-client metrics. Horizontal Federated Learning is **the foundation of practical federated systems** — by enabling organizations with compatible data schemas to collaborate without centralizing data, it makes privacy-preserving machine learning at scale a reality, powering applications from mobile keyboards to healthcare to financial services.