load balancing (moe)

Load balancing in MoE ensures experts are used roughly equally, preventing underutilization and bottlenecks. **The problem**: Without balancing, router may send most tokens to few experts. Others underutilized, those overloaded become bottlenecks. **Consequences of imbalance**: Wasted parameters (unused experts), computation bottlenecks (overused experts), reduced effective capacity. **Auxiliary loss**: Add loss term penalizing imbalanced usage. Encourages router to spread tokens evenly. Loss proportional to variance of expert loads. **Capacity factor**: Set maximum tokens per expert (e.g., 1.25x fair share). Excess tokens dropped or rerouted. **Expert choice routing**: Let experts choose tokens rather than tokens choosing experts. Guarantees balance. **Implementation challenges**: Balance per-batch, per-sequence, or globally. Trade-offs with routing quality. **Switch Transformer approach**: Top-1 routing with capacity factor and aux loss. **Current best practices**: Combine auxiliary loss with capacity factors. Tune balance between routing quality and load balance. **Monitoring**: Track expert utilization during training. Imbalance indicates routing or loss tuning issues.