Heterogeneous Memory Management is the hardware and software infrastructure that provides a unified virtual address space across CPUs, GPUs, and other accelerators — enabling automatic data migration between device memories based on access patterns, eliminating manual memory allocation and transfer management from the programmer's responsibility.

Unified Virtual Addressing (UVA):

• Single Address Space: CPU and GPU share a common 48-bit virtual address space; any pointer is valid on both devices, and the runtime can determine the physical location from the address — eliminates separate cudaMalloc/malloc allocations
• Managed Memory (cudaMallocManaged): allocates memory accessible from both CPU and GPU; the CUDA runtime automatically migrates pages to the accessing processor on demand via page faults
• Page Fault Migration: when a GPU thread accesses a page residing in CPU memory, the GPU MMU generates a page fault; the driver migrates the 64KB page to GPU memory (or maps it remotely via NVLink); subsequent accesses hit local memory at full bandwidth
• Prefetch Hints: cudaMemPrefetchAsync moves pages proactively before access — avoiding page fault latency (10-100 μs per fault); essential for performance-critical code paths

Migration Policies:

• First-Touch Migration: page migrates to the processor that first accesses it; optimal for producer-consumer patterns where one processor writes and another reads sequentially
• Access Counter Migration: hardware access counters track frequency of remote accesses; pages exceeding a threshold migrate to the primary accessor — prevents thrashing for shared data
• Read-Duplication: read-only pages can be replicated across multiple GPU memories, allowing all GPUs to read at local bandwidth; write access invalidates copies and migrates the single writable copy
• Pinned/Non-Migratable: critical data structures (page tables, DMA buffers) are pinned to specific memories; cudaMemAdvise(cudaMemAdviseSetAccessedBy) hints the runtime to place pages optimally without migration

Multi-GPU Memory:

• Peer-to-Peer Access: GPUs connected via NVLink can access each other's memory directly without CPU involvement; latency ~1-2 μs vs ~10 μs for PCIe; bandwidth 300-900 GB/s bidirectional per NVLink connection
• System Memory Mapping: GPU can map and access CPU system memory at reduced bandwidth (~32 GB/s via PCIe Gen5); useful for large datasets that exceed GPU memory
• Memory Oversubscription: managed memory enables GPU computations on datasets larger than GPU physical memory by transparently evicting and fetching pages; performance degrades gracefully rather than failing with out-of-memory
• CXL Memory Expansion: emerging CXL-attached memory pools extend the unified address space to disaggregated memory with ~200-400 ns latency from CPU perspective

Performance Optimization:

• Avoid Thrashing: CPU and GPU alternately accessing the same pages causes repeated migration — restructure algorithms for phase-based access (GPU phase, CPU phase) with prefetch at phase boundaries
• Large Page Support: 2MB huge pages reduce page table overhead and migration frequency — fewer faults for sequential access patterns; enabled via cudaMemAdvise
• Stream-Ordered Allocation: cudaMallocAsync/cudaFreeAsync allocate from per-stream memory pools, enabling efficient temporary allocation without synchronization overhead

Heterogeneous memory management is the programming model evolution that transforms GPU computing from explicit memory management (cudaMemcpy everywhere) to transparent data access — enabling productivity comparable to shared-memory programming while preserving the performance benefits of data locality through intelligent automatic migration.