numa architecture memory access

**NUMA Architecture and Memory Affinity** enable **explicit placement of data and threads on multi-socket systems to exploit local memory bandwidth and latency, critical for HPC and data-center applications scaling to 100s of cores.** **Non-Uniform Memory Access Topology** - **NUMA Organization**: Multiple sockets (CPUs), each with local memory attached. Local socket memory ~100ns latency, remote socket memory ~200-400ns (2-4x penalty). - **Memory Bandwidth Asymmetry**: Local DRAM bandwidth (say 100 GB/s) shared with other local cores. Remote DRAM bandwidth crossed via QPI/Infinity Fabric interconnect (less bandwidth than local). - **Example Topology**: Dual-socket Xeon with 32 cores per socket. Each core can access both socket's memory, but local access preferred. - **UMA vs NUMA**: Older systems uniform memory access (UMA) via shared front-side bus. Modern systems inherently NUMA due to scaling limitations of centralized memory controller. **NUMA Node Binding and Thread Affinity** - **NUMA Node Definition**: Logical grouping of cores + associated memory. Socket-based binding: threads pinned to cores in same socket as their data. - **numactl Command**: numactl --membind=node0 --cpunodebind=node0 application. Forces threads/memory to specific NUMA node. Prevents OS migration. - **libnuma Library**: Programmatic NUMA control. numa_alloc_onnode(), numa_bind(), numa_set_preferred(). Enables application-level NUMA awareness. - **cpuset Cgroups**: Linux control groups restrict processes to CPU/memory subsets. System-wide NUMA orchestration via cgroups. **First-Touch Policy** - **Memory Allocation Mechanism**: Pages allocated to NUMA node of thread first accessing page (write). OS tracks page residency. - **Default Behavior**: malloc() allocates from kernel's allocator, typically interleaved across nodes (round-robin). Application overrides via numa_alloc_onnode(). - **First-Touch Implication**: Thread A allocates array B but doesn't initialize; Thread B initializes B. B ends up on B's node (correct affinity). - **Guideline**: Initialize data on thread that will access it, or explicitly allocate on target node before other threads touch. **Remote vs Local Memory Latency Impact** - **Latency Difference**: Local ~100ns, remote ~300ns (3x penalty). Impacts iterative workloads (large loop counts × remote access = significant slowdown). - **Bandwidth Scaling**: Remote bandwidth congested by all-to-all access patterns. Single-socket bandwidth ~100 GB/s; multi-socket aggregate ~150-200 GB/s (sub-linear). - **Cache Effects**: L3 cache (8-20 MB per socket) mitigates some remote access penalties. If working set fits in L3, remote penalty minimal. - **Example Impact**: 1000-iteration loop accessing remote memory: 1000 × 200ns = 200µs (remote) vs 100µs (local). 2x slowdown possible. **NUMA-Aware Data Structures** - **Replicated Data**: Hot data replicated per socket (each socket has copy). Slight memory overhead but eliminates remote access. - **Data Partitioning**: Divide large arrays by NUMA node. Thread i processes array[i×partition_size:(i+1)×partition_size]. Guarantees local access. - **Hash Table Striping**: Hash table buckets assigned to NUMA nodes. Hash function distributes keys across nodes balancing load and access locality. - **Graph Partitioning**: Graph algorithms (matrix computations, machine learning) partition vertices/edges by NUMA locality. Minimize cross-node edges. **Memory Interleaving vs Binding** - **Interleaved Mode**: OS spreads pages round-robin across NUMA nodes. Balances memory usage but serializes remote access across all nodes. Poor latency. - **Bound Mode**: Pages allocated on specific node. Requires explicit NUMA awareness (application or numactl). Excellent latency but requires work distribution matching binding. - **Hybrid Approaches**: Bind hot/critical data to local node, interleave cold data. Best of both worlds. **NUMA Scheduling and OS Coordination** - **OS NUMA Scheduler**: Linux kernel scheduler (CFS) considers NUMA locality. Migrates threads toward memory (if cheaper than migrating memory). - **Task Scheduler Trade-offs**: Migrate thread (cache cold) vs keep thread (remote memory). Decision based on current load, task runtime, memory intensity. - **AutoNUMA**: Linux feature periodically migrates pages toward threads that access them (and vice versa). Reduces manual tuning but adds overhead. **NUMA in Multi-Socket HPC Servers** - **Dual/Quad Socket Systems**: 2-4 sockets per server, 64-256 cores total. Typical HPC configuration in data centers. - **Binding Strategy**: MPI ranks bound to NUMA nodes (one rank per node). Inter-rank communication via network (InfiniBand) not NUMA crossings. - **Memory Scaling**: Dual-socket Xeon: 256 GB-1 TB memory (128GB-512GB per socket). Single-node jobs fit; larger jobs spill to other nodes (network-based, slower). - **Benchmark Sensitivity**: STREAM benchmark 5-10x slower on remote nodes vs local. Gemm (compute-bound) largely unaffected by NUMA.