GPU Virtual Memory and Memory Management is the system software and hardware infrastructure that provides address translation, demand paging, and memory protection for GPU computations — enabling unified virtual addressing (UVA) across CPU and GPU, memory oversubscription (GPU programs accessing more memory than physically available on the GPU), and coherent shared memory between CPU and GPU through hardware page fault handling, fundamentally simplifying GPU programming for large-dataset workloads.

Traditional GPU Memory Model

Before unified memory, programmers explicitly managed two separate address spaces: 1. Allocate on CPU: malloc() or new 2. Allocate on GPU: cudaMalloc() 3. Copy CPU→GPU: cudaMemcpy(dst_gpu, src_cpu, size, HostToDevice) 4. Launch kernel on GPU data 5. Copy GPU→CPU: cudaMemcpy(dst_cpu, src_gpu, size, DeviceToHost)

This explicit management is error-prone, verbose, and prevents data structures with pointers from being shared between CPU and GPU (pointers are address-space-specific).

Unified Virtual Addressing (UVA)

CUDA 4.0+ provides a single virtual address space shared by CPU and all GPUs:

• Every pointer uniquely identifies its location (CPU, GPU 0, GPU 1, ...).
• cudaMemcpy can determine copy direction from pointer addresses — no need to specify HostToDevice/DeviceToHost.
• Pointers can be passed between CPU and GPU functions, enabling shared data structures.

Managed Memory (cudaMallocManaged)

CUDA Unified Memory allocates memory accessible by both CPU and GPU:

• The runtime automatically migrates pages between CPU and GPU on access.
• First-touch policy: pages are physically allocated where first accessed.
• Hardware page faults (Pascal+): when GPU accesses a page resident on CPU, a GPU page fault triggers automatic migration. No programmer intervention.
• Prefetch hints: cudaMemPrefetchAsync() migrates pages proactively, avoiding fault latency.

GPU Page Fault Hardware

NVIDIA Pascal and later GPUs include a hardware page fault handler:

• Fault Detection: GPU MMU detects access to non-resident or non-mapped pages and raises a fault.
• Fault Handling: GPU fault handler traps to the driver, which (1) maps the page from CPU to GPU, (2) migrates the data, and (3) updates the GPU page table. The faulting warp is stalled during migration; other warps continue executing.
• Latency: Page fault + migration: 20-100 μs (dominated by PCIe transfer for 4KB-2MB pages). Much slower than a TLB miss (~100 ns).

Memory Oversubscription

GPU physical memory is limited (24-80 GB). With page faults, GPU programs can address more memory than physically available — excess pages are evicted to CPU memory and fetched on demand. Enables running problems larger than GPU memory without manual data management. Performance degrades gracefully with oversubscription ratio.

Multi-GPU Memory

• Peer Access: GPUs connected via NVLink can directly access each other's memory without CPU involvement. cudaMemcpyPeer() or direct load/store with UVA.
• NVSwitch Full Connectivity: All GPUs in an NVLink domain (DGX H100: 8 GPUs) can access all other GPUs' memory at full NVLink bandwidth (900 GB/s per GPU).
• CUDA Memory Pools: cudaMallocAsync() and stream-ordered memory allocation enable efficient memory reuse without explicit free/realloc cycles.

GPU Virtual Memory and Memory Management is the system infrastructure that evolves GPU programming from explicit buffer management to transparent shared memory — enabling the programming simplicity of unified addressing while providing the hardware mechanisms for efficient data migration between CPU and GPU memory.