mixture of experts efficient inference

**MoE Inference Optimization: Sparse Expert Activation — achieving throughput scaling without proportional latency increase** Mixture of Experts (MoE) models like Mixtral 8x7B activate only subset of experts per token, enabling large model capacity with controlled inference cost. Deployment optimization focuses on expert routing, load balancing, and memory management. **Expert Selection and Load Imbalance** Router network: per-token scalar output per expert, softmax selects top-k (usually k=2). Token routes to 2 experts; only 2/64 (3%) experts compute per token. Load imbalance challenge: some tokens route to same expert cluster (e.g., all Spanish tokens to Spanish-expert group), causing uneven load across TPU/GPU clusters. Solution: auxiliary loss encouraging balanced routing (add small regularization pushing load distribution toward uniform). **Expert Affinity and Token Clustering** Tokens of similar meaning route to same experts across layers (expert affinity). Utilization insight: don't just activate random experts; learn which experts specialize in which domains. Communication pattern: only active experts' outputs required per layer—activate 20% experts, transfer 20% weights per layer (vs. 100% for dense models). Clustering: similar tokens activate similar experts → sequential access pattern (cache-friendly). **Expert Caching and Memory Hierarchy** Expert weights stored: HBM (high-bandwidth memory on GPU) or off-chip (CPU DRAM, network storage). Bottleneck: loading expert weights into GPU compute dies. Solution: multi-level cache (reserved HBM buffer for hot experts). Prediction: given token, predict which experts activate; prefetch weights into HBM. Cooperative prefetching: batch multiple token routing decisions, amortize prefetch overhead. Trade-off: larger cache (more HBM) reserves capacity, reducing KV-cache for context (longer context = less HBM available for experts). **Batch Routing and Grouping** Naive batching: heterogeneous routing (different tokens route to different experts) complicates GPU scheduling (idle warps). Solution: group tokens by activated expert set, fuse kernels. All-to-all communication (AllReduce) after local expert computation gathers results. Cost: communication can dominate for sparse activation if batch size small. **Throughput vs. Latency Tradeoff** Dense models (GPT-3.5): lower latency (single forward pass, no routing overhead). MoE (Mixtral): lower per-token latency (fewer compute ops) but routing overhead (network latency, load imbalance stalls). Throughput: MoE achieves higher throughput (more tokens per second across cluster) due to lower compute per token. Single-token latency: often higher in MoE vs. dense (batch size 1, routing overhead dominates). Inference serving: batch requests together to amortize routing overhead; disaggregate experts across dedicated workers (expert parallelism) to hide load imbalance.