network on chip noc architecture

**Network-on-Chip NoC Architecture** — Network-on-chip (NoC) architectures replace traditional bus-based and crossbar interconnects with packet-switched communication networks, providing scalable, high-bandwidth on-chip data transport that supports the growing number of processing elements in modern system-on-chip designs. **NoC Topology Design** — Network structure determines communication characteristics: - Mesh topologies arrange routers in regular two-dimensional grids with nearest-neighbor connections, providing predictable latency, balanced bandwidth, and straightforward physical implementation - Ring and torus topologies connect routers in circular configurations with optional wrap-around links that reduce maximum hop count at the cost of longer physical wire lengths - Tree and fat-tree topologies provide hierarchical bandwidth aggregation suitable for memory subsystem interconnects where traffic patterns converge toward shared resources - Irregular and application-specific topologies optimize connectivity for known communication patterns, eliminating unnecessary links to reduce area and power overhead - Heterogeneous NoC architectures combine different topology segments — high-bandwidth meshes for compute clusters with low-latency rings for control traffic — within a single chip **Router Architecture and Microarchitecture** — NoC routers perform packet switching and forwarding: - Input-buffered router architectures store incoming flits in per-port FIFO buffers, with virtual channels multiplexing multiple logical channels onto each physical link - Pipeline stages including buffer write, route computation, virtual channel allocation, switch allocation, and switch traversal determine single-hop router latency - Crossbar switch fabrics connect input ports to output ports based on arbitration decisions, with full crossbar designs supporting simultaneous non-conflicting transfers - Wormhole flow control divides packets into flits that traverse the network in pipeline fashion, reducing buffer requirements compared to store-and-forward - Credit-based flow control mechanisms prevent buffer overflow by regulating flit injection rates based on downstream availability **Routing and Flow Control** — Algorithms determine packet paths through the network: - Deterministic routing (XY routing in meshes) sends all packets between a source-destination pair along identical paths, simplifying implementation but potentially creating hotspots - Adaptive routing algorithms dynamically select paths based on network congestion, distributing traffic more evenly at the cost of increased router complexity and potential out-of-order delivery - Deadlock avoidance through virtual channel allocation, turn restrictions, or escape channels prevents circular dependencies that would stall traffic - Source routing embeds the complete path in packet headers, eliminating route computation at intermediate routers - Multicast and broadcast support enables efficient one-to-many communication for cache coherence protocols and synchronization **Quality of Service and Performance** — NoC design targets application requirements: - Traffic class prioritization assigns different service levels to latency-sensitive control traffic versus bandwidth-intensive data transfers - Bandwidth reservation through time-division multiplexing provides deterministic throughput for real-time processing elements - End-to-end latency optimization minimizes hop count, router pipeline depth, and serialization delay for critical paths - Power management techniques including clock gating idle routers, dynamic voltage scaling of network segments, and power-gating unused links reduce NoC energy consumption **Network-on-chip architecture provides the scalable communication backbone essential for modern multi-core and heterogeneous SoC designs, where interconnect bandwidth and latency increasingly determine overall system performance.** --- **AI Accelerator Architecture — Compute, Memory, and Interconnect.** Modern AI chips are purpose-built for matrix multiplication: a systolic array or tensor core computes thousands of multiply-accumulate (MAC) operations per cycle, fed by a memory hierarchy (registers → SRAM → HBM) connected through a network-on-chip (NoC) that determines whether the compute units starve or stay busy. The single metric that captures this interaction is the roofline model: peak performance (TFLOPS) vs memory bandwidth (TB/s), where the arithmetic intensity of the workload (FLOPs/byte) determines which resource limits throughput. **Tensor Cores — The Matrix Multiply Unit.** NVIDIA tensor cores perform 4$\times$4 matrix multiply-accumulate (D = A$\times$B + C) in a single clock cycle at mixed precision (FP16 inputs, FP32 accumulate). The H100 has 528 tensor cores across 132 SMs, delivering 989 TFLOPS at FP16 or 1,979 TFLOPS at FP8 — a 3$\times$ generational improvement over A100 (312 TFLOPS FP16). Programming tensor cores requires structuring data in tile-friendly layouts (16$\times$16 or 32$\times$8 fragments) via CUDA WMMA or MMA PTX instructions. Utilization typically reaches 60–80% in production training (compute-bound GEMM) but drops to 10–30% during inference decode (memory-bound, limited by KV cache reads). AMD CDNA3 Matrix Cores and Google TPU v5 MXUs provide equivalent functionality at comparable TFLOPS/W. **KV Cache and Inference Efficiency.** During autoregressive LLM inference, each generated token requires reading the full key-value cache of all prior tokens — creating a memory-bandwidth bottleneck where arithmetic intensity drops to 1–5 FLOPs/byte (far left of the roofline). A 70B-parameter model at sequence length 4096 stores 40 GB of KV cache in HBM; generating each token reads 40 GB at 3.35 TB/s = 12 ms latency per token — regardless of compute capacity. Solutions: PagedAttention (vLLM) eliminates KV cache fragmentation; multi-query attention (MQA/GQA) reduces KV size by 8$\times$; speculative decoding verifies 4–8 draft tokens per forward pass, increasing effective throughput 2–4$\times$; continuous batching (Orca) amortizes KV reads across multiple sequences in flight. **Network-on-Chip (NoC) for AI Accelerators.** The NoC connects hundreds of compute tiles (tensor cores, memory controllers, I/O ports) through a mesh, ring, or hierarchical topology — and its bisection bandwidth determines the maximum data rate for all-reduce operations during distributed training. An H100 has a 12$\times$11 crossbar connecting 132 SMs, 6 HBM3 stacks, and 18 NVLink ports. The total internal bandwidth exceeds 30 TB/s. For multi-chip training, NVLink 4.0 provides 900 GB/s chip-to-chip (18 links $\times$ 50 GB/s each) while PCIe 5.0 adds 128 GB/s for host communication. The NoC design determines whether the GPU can keep all tensor cores fed during a 2048-GPU training run where each iteration requires an all-reduce of 1–10 GB of gradients across the fabric. **Mixture of Experts (MoE) — Hardware Implications.** MoE models (GPT-4, Mixtral, Switch Transformer) activate only 2–8 experts per token out of 64–256 total, reducing compute by 10–30$\times$ relative to a dense model of equivalent capacity — but at the cost of massive memory footprint (every expert's weights must reside in HBM) and irregular memory access patterns that stress the NoC and memory controller. A Mixtral 8$\times$7B model has 46.7B total parameters but only 12.9B active per token; the challenge is that expert routing is data-dependent and unpredictable, causing load imbalance across GPU SMs and across nodes in distributed inference. Hardware solutions include expert parallelism (each GPU holds a subset of experts), capacity factors limiting expert overload, and all-to-all communication patterns that require high bisection bandwidth.