heterogeneous memory management
**Heterogeneous Memory Management** is **the hardware and software infrastructure that provides a unified virtual address space across CPUs, GPUs, and other accelerators — enabling automatic data migration between device memories based on access patterns, eliminating manual memory allocation and transfer management from the programmer's responsibility**.
**Unified Virtual Addressing (UVA):**
- **Single Address Space**: CPU and GPU share a common 48-bit virtual address space; any pointer is valid on both devices, and the runtime can determine the physical location from the address — eliminates separate cudaMalloc/malloc allocations
- **Managed Memory (cudaMallocManaged)**: allocates memory accessible from both CPU and GPU; the CUDA runtime automatically migrates pages to the accessing processor on demand via page faults
- **Page Fault Migration**: when a GPU thread accesses a page residing in CPU memory, the GPU MMU generates a page fault; the driver migrates the 64KB page to GPU memory (or maps it remotely via NVLink); subsequent accesses hit local memory at full bandwidth
- **Prefetch Hints**: cudaMemPrefetchAsync moves pages proactively before access — avoiding page fault latency (10-100 μs per fault); essential for performance-critical code paths
**Migration Policies:**
- **First-Touch Migration**: page migrates to the processor that first accesses it; optimal for producer-consumer patterns where one processor writes and another reads sequentially
- **Access Counter Migration**: hardware access counters track frequency of remote accesses; pages exceeding a threshold migrate to the primary accessor — prevents thrashing for shared data
- **Read-Duplication**: read-only pages can be replicated across multiple GPU memories, allowing all GPUs to read at local bandwidth; write access invalidates copies and migrates the single writable copy
- **Pinned/Non-Migratable**: critical data structures (page tables, DMA buffers) are pinned to specific memories; cudaMemAdvise(cudaMemAdviseSetAccessedBy) hints the runtime to place pages optimally without migration
**Multi-GPU Memory:**
- **Peer-to-Peer Access**: GPUs connected via NVLink can access each other's memory directly without CPU involvement; latency ~1-2 μs vs ~10 μs for PCIe; bandwidth 300-900 GB/s bidirectional per NVLink connection
- **System Memory Mapping**: GPU can map and access CPU system memory at reduced bandwidth (~32 GB/s via PCIe Gen5); useful for large datasets that exceed GPU memory
- **Memory Oversubscription**: managed memory enables GPU computations on datasets larger than GPU physical memory by transparently evicting and fetching pages; performance degrades gracefully rather than failing with out-of-memory
- **CXL Memory Expansion**: emerging CXL-attached memory pools extend the unified address space to disaggregated memory with ~200-400 ns latency from CPU perspective
**Performance Optimization:**
- **Avoid Thrashing**: CPU and GPU alternately accessing the same pages causes repeated migration — restructure algorithms for phase-based access (GPU phase, CPU phase) with prefetch at phase boundaries
- **Large Page Support**: 2MB huge pages reduce page table overhead and migration frequency — fewer faults for sequential access patterns; enabled via cudaMemAdvise
- **Stream-Ordered Allocation**: cudaMallocAsync/cudaFreeAsync allocate from per-stream memory pools, enabling efficient temporary allocation without synchronization overhead
Heterogeneous memory management is **the programming model evolution that transforms GPU computing from explicit memory management (cudaMemcpy everywhere) to transparent data access — enabling productivity comparable to shared-memory programming while preserving the performance benefits of data locality through intelligent automatic migration**.