Mixture of Experts - Conditional Computation

# Mixture of Experts - Conditional Computation

## Introduction & Motivation

Mixture of Experts: route tokens to specialized experts. Conditional computation for efficiency. Applications: scalable models, specialized processing.

Motivation: Enable selective computation for efficiency and specialization.

Applications: Large-scale models, efficient scaling.

---

## Core Concepts & Theory

### Gating Function

Route tokens to experts.

### Load Balancing

Prevent expert overload.

### Expert Specialization

Learn specialized skills.

### Conditional Computation

Activate subset of parameters.

---

## Mathematical Formulation

Gating Output:
$$y = \sum_i g(x)_i E_i(x)$$

Where g is gating, E_i are experts

Load Balance Loss:
$$\mathcal{L}_{ ext{balance}} = ext{coefficient} \cdot ext{cv}(P_i)^2$$

---

## Advanced Theory & Extensions

### Expert Routing

Sophisticated gating strategies.

### Load Balancing

Prevent collapse.

### Expert Dropout

Regularize during training.

---

## Computational Considerations

Experts: Active subset.

Gating: O(d·e) per token.

Scaling: Efficient with experts.

---

## Practical Implementation Strategies

### Auxiliary Loss

Encourage load balancing.

### Capacity Factors

Control expert capacity.

### Top-K Gating

Route to top experts.

---

## Benchmark Datasets & Evaluation

Language Modeling: Perplexity.

Machine Translation: BLEU.

Vision: ImageNet accuracy.

---

## Key Challenges & Limitations

### Load Imbalance

Experts may be underutilized.

### Training Instability

Requires careful balancing.

### Communication

Distributed training complexity.

---

## Hyperparameter Tuning

Number of experts: 32-2048.

Expert capacity: 1.5-2.0.

Top-K: 2-4.

---

## Real-World Applications & Case Studies

Large Scale: Switch Transformers.

Vision: ViT-MoE.

Multimodal: MUSE architecture.

---

## Integration with Other Methods

MoE + load balancing; + efficiency tricks.

---

## Summary & Key Takeaways

MoE enables efficient large-scale models.

Principles:
1. Gating: Route tokens.
2. Experts: Specialized modules.
3. Scaling: O(e) not O(1).
4. Load balancing: Critical.
5. Efficiency: Compute only needed paths.

---

## Appendix: Practical Labs

### Lab 1: Gating Function

import numpy as np

def moe_gating(x, num_experts=8, top_k=2):
 """MoE gating function"""
 # Gate logits
 gate_logits = x @ np.random.randn(768, num_experts)
 gate_probs = np.exp(gate_logits) / np.sum(np.exp(gate_logits), axis=1, keepdims=True)
 
 # Top-k selection
 top_indices = np.argsort(gate_probs)[:, -top_k:]
 top_probs = np.take_along_axis(gate_probs, top_indices, axis=1)
 
 return top_indices, top_probs

np.random.seed(42)
x = np.random.randn(10, 768)
indices, probs = moe_gating(x, 8, 2)
assert indices.shape == (10, 2)
print("✓ Gating function working")

### Lab 2: Load Balancing Loss

import numpy as np

def compute_load_balance_loss(gate_probs, num_experts):
 """Compute load balance auxiliary loss"""
 # Average probability per expert
 mean_probs = np.mean(gate_probs, axis=0)
 
 # Coefficient of variation
 cv = np.std(mean_probs) / (np.mean(mean_probs) + 1e-8)
 
 balance_loss = cv ** 2
 return balance_loss

np.random.seed(42)
probs = np.random.dirichlet([1]*8, 100)
loss = compute_load_balance_loss(probs, 8)
assert loss >= 0
print(f"✓ Load balance loss: {loss:.4f}")

### Lab 3: Expert Output Aggregation

import numpy as np

def aggregate_expert_outputs(expert_outputs, gate_indices, gate_probs):
 """Aggregate outputs from routed experts"""
 batch_size = len(gate_indices)
 output_dim = expert_outputs[0].shape[-1]
 aggregated = np.zeros((batch_size, output_dim))
 
 for i in range(batch_size):
 for k, (expert_id, prob) in enumerate(zip(gate_indices[i], gate_probs[i])):
 aggregated[i] += prob * expert_outputs[expert_id][i]
 
 return aggregated

np.random.seed(42)
expert_outputs = [np.random.randn(10, 768) for _ in range(8)]
gate_indices = np.array([[0, 3], [1, 5], [2, 7]] * 3 + [[0, 1]])
gate_probs = np.random.dirichlet([1]*2, 10)
agg = aggregate_expert_outputs(expert_outputs, gate_indices, gate_probs)
assert agg.shape == (10, 768)
print("✓ Expert aggregation working")

### Lab 4: Expert Utilization

import numpy as np

def compute_expert_utilization(gate_indices, num_experts):
 """Compute expert utilization"""
 utilization = np.zeros(num_experts)
 
 for indices in gate_indices:
 for expert_id in indices:
 utilization[expert_id] += 1
 
 utilization = utilization / len(gate_indices)
 
 return utilization

gate_indices = np.random.randint(0, 8, (100, 2))
utilization = compute_expert_utilization(gate_indices, 8)
assert len(utilization) == 8
assert np.allclose(utilization.sum(), 2.0)
print(f"✓ Utilization: {utilization.mean():.2f} avg per expert")

---