Router Networks are the specialized routing components in Mixture-of-Experts (MoE) architectures that assign tokens to expert sub-networks across distributed computing devices, managing the physical data movement (all-to-all communication) required when tokens on one GPU need to be processed by experts residing on different GPUs — the systems engineering layer that transforms the logical routing decisions of gating networks into efficient hardware-level data transfers across the interconnect fabric of large-scale model serving infrastructure.

What Are Router Networks?

• Definition: A router network extends the gating network concept to the distributed systems domain. While a gating network computes which expert should process each token, the router network handles the physical mechanics — buffering tokens, communicating routing decisions across devices, executing all-to-all data transfers, managing expert capacity constraints, and handling token overflow when more tokens are assigned to an expert than its buffer can hold.
• All-to-All Communication: In a distributed MoE model where each GPU hosts a subset of experts, routing tokens to their assigned experts requires all-to-all communication — every device sends some tokens to every other device and receives some tokens from every other device. This collective operation is the primary communication bottleneck in MoE inference and training.
• Capacity Factor: Each expert has a fixed buffer size (capacity) that limits how many tokens it can process per forward pass. The capacity factor $C$ (typically 1.0–1.5) determines the buffer size as $C imes (N_{tokens} / N_{experts})$. Tokens that exceed an expert's capacity are dropped (not processed) and use only the residual connection, losing information.

Why Router Networks Matter

• Scalability Bottleneck: The all-to-all communication pattern scales with the product of sequence length and number of devices. At the scale of GPT-4-class models serving millions of requests, the router's communication efficiency directly determines whether the MoE architecture delivers its theoretical efficiency gains or is bottlenecked by inter-device data movement.
• Token Dropping: When routing is imbalanced (many tokens assigned to popular experts, few to unpopular ones), tokens are dropped at capacity-constrained experts. Dropped tokens bypass expert processing entirely, receiving only the residual connection — potentially degrading output quality. Router design must minimize dropping through balanced routing.
• Expert Parallelism: Router networks enable expert parallelism — distributing experts across devices so that each device processes different experts in parallel. This parallelism strategy is complementary to data parallelism (same model, different data) and tensor parallelism (same layer split across devices), forming the third axis of large-model parallelism.
• Latency vs. Throughput: Router networks must balance latency (time for a single token to traverse the routing and expert processing pipeline) against throughput (total tokens processed per second). Batching tokens for efficient all-to-all communication improves throughput but increases latency — a trade-off that must be tuned for the deployment scenario.

Router Network Challenges

Router Networks are the hardware packet switches of neural computation — managing the physical movement of data chunks between specialized expert modules across distributed computing infrastructure, ensuring that the theoretical efficiency of conditional computation is realized in practice despite the communication costs of large-scale distributed systems.