sparse model

Sparse models activate only a subset of parameters for each input, enabling larger total capacity with fixed compute. **Core idea**: Route each input to subset of model (experts), rest of parameters inactive. More total parameters without proportional compute increase. **Mixture of Experts (MoE)**: Predominant sparse architecture. Router selects which experts process each token. **Sparsity patterns**: Expert-based (MoE), unstructured sparsity (zero weights), attention sparsity (attend to subset of tokens). **Efficiency gain**: 8x7B MoE has 56B total params but activates only 7B per token. Compute of 7B, capacity approaching 56B. **Training challenges**: Load balancing (experts used equally), routing stability, communication overhead in distributed training. **Inference considerations**: Need all parameters in memory even if not all active. Different compute vs memory trade-off than dense. **Examples**: Mixtral 8x7B, GPT-4 (rumored), Switch Transformer, GShard. **Advantages**: Scale capacity without proportional compute, potential for specialization. **Disadvantages**: More complex, less predictable, some routing overhead. Increasingly important for frontier models.