gating network

A gating network (also called a router) is the component in Mixture of Experts (MoE) architectures that determines which expert networks should process each input token, enabling sparse conditional computation by routing different inputs to different specialized subnetworks. The gating network is critical to MoE performance — it must learn to assign tokens to the most appropriate experts while maintaining balanced utilization across all experts. The basic gating mechanism works as follows: given an input token representation x with hidden dimension d, the gating network computes scores for each expert using a learned linear projection: g(x) = softmax(W_g · x), where W_g is a trainable matrix of shape (num_experts × d_model). The top-k experts with the highest scores are selected (typically k=1 or k=2), and the output is the weighted sum of selected expert outputs: y = Σ g_i(x) · Expert_i(x) for selected experts i. Gating network designs include: top-k gating (selecting the k highest-scored experts per token — Switch Transformer uses k=1, Mixtral uses k=2), noisy top-k (adding calibrated noise before selection to encourage exploration during training — preventing early expert specialization), expert choice routing (experts select tokens rather than tokens selecting experts — ensuring perfect load balance), hash routing (deterministic assignment based on token hashing — eliminating the learned router entirely), and soft routing (all experts process every token with soft attention weights — dense but differentiable). Load balancing is the central challenge: without explicit balancing mechanisms, the gating network tends to collapse — sending most tokens to a few "winner" experts while others receive little training signal and atrophy. Balancing strategies include auxiliary load-balancing losses (penalizing uneven expert utilization), capacity factors (limiting the maximum number of tokens per expert), and batch-level priority routing. The gating network typically adds negligible parameters (a single linear layer) but fundamentally determines the efficiency and quality of the entire MoE model.