numa aware programming

**NUMA-Aware Programming** is **the practice of structuring parallel applications to account for the non-uniform memory access costs of modern multi-socket systems — placing data in memory local to the processors that access it and scheduling threads to cores near their data, achieving 2-4× performance improvement over NUMA-oblivious approaches for memory-bandwidth-sensitive workloads**. **NUMA Architecture:** - **Multi-Socket Topology**: each CPU socket has local DRAM channels providing ~200-400 GB/s bandwidth; accessing remote DRAM on another socket traverses inter-socket links (UPI, Infinity Fabric) with 1.5-3× higher latency and reduced bandwidth - **NUMA Nodes**: each socket (or sub-socket on large processors) forms a NUMA node with its own memory controller; topology is exposed via /sys/devices/system/node on Linux and queried via hwloc or numactl - **Distance Matrix**: NUMA distances quantify relative access costs; local access = distance 10 (reference); cross-socket = distance 20-32; cross-NUMA within one socket (sub-NUMA clustering) = distance 12-16 - **Memory Interleaving**: default Linux policy interleaves pages across NUMA nodes for average-case performance; dedicated applications benefit from explicit NUMA-local allocation **Memory Allocation Policies:** - **First-Touch**: Linux default for private allocations; page is allocated on the NUMA node where the first page fault occurs — initialization thread determines placement; parallel first-touch (each thread initializes its portion) distributes pages correctly - **numactl --membind/--interleave**: command-line control of NUMA policy; --membind=N restricts allocation to node N; --interleave=0,1 distributes pages round-robin for shared data accessed by all sockets equally - **mbind/set_mempolicy**: programmatic NUMA policy control at page granularity; MPOL_BIND forces allocation on specified nodes; MPOL_PREFERRED suggests a node but falls back if memory is unavailable; MPOL_INTERLEAVE distributes evenly - **Huge Pages**: 2MB and 1GB huge pages reduce TLB misses and improve memory access predictability; NUMA-local huge page allocation requires explicit reservation (hugetlbfs) or transparent huge pages (THP) with NUMA awareness **Thread-Data Affinity:** - **CPU Pinning**: pthread_setaffinity_np or taskset binds threads to specific cores; ensuring thread i runs on the same NUMA node as its data partition eliminates cross-socket memory access - **OpenMP Affinity**: OMP_PLACES=cores and OMP_PROC_BIND=close/spread control thread placement; close packing fills one socket before using the next (good for memory-intensive, socket-local workloads); spread distributing evenly across sockets maximizes aggregate bandwidth - **Work Partitioning**: divide data arrays so that each NUMA node owns a contiguous chunk; assign threads on each node to process their local chunk; reduction operations across nodes use a two-level hierarchy (local reduce, then cross-node reduce) - **Migration Detection**: Linux AutoNUMA (NUMA balancing) periodically unmaps pages and remaps them on the accessing node when consistent cross-node access is detected — automatic but introduces TLB shootdown overhead **Performance Diagnosis:** - **perf stat -e numa-***: hardware performance counters track local vs remote memory accesses; remote access ratio >20% indicates NUMA placement issues for bandwidth-sensitive code - **numastat**: reports per-node memory allocation statistics; large numa_miss counts indicate first-touch allocation on wrong nodes — initialization pattern needs correction - **Memory Bandwidth Measurement**: STREAM benchmark per-node measures local bandwidth capacity; cross-node bandwidth is typically 30-50% of local — the NUMA penalty quantifies the optimization opportunity - **Intel VTune / AMD uProf**: visualize NUMA access patterns and identify hot data structures causing cross-socket traffic; guide data layout reorganization and thread pinning decisions NUMA-aware programming is **essential for achieving peak performance on modern multi-socket servers — the 2-3× bandwidth difference between local and remote memory access means that memory placement and thread affinity decisions have a first-order impact on application throughput, especially for memory-bandwidth-bound HPC, database, and machine learning workloads**.