gpu memory management virtual

**GPU Virtual Memory Management** is the **system of hardware and software mechanisms that provide GPUs with virtual address spaces, demand paging, memory oversubscription, and unified addressing** — evolving GPU memory from simple physical allocation to sophisticated virtual memory systems comparable to CPU memory management. Historically, GPU memory was managed as a simple physical allocator: applications allocated fixed-size buffers in GPU VRAM, and any overflow required manual data staging through host memory. Modern GPUs provide full virtual memory support that fundamentally changes programming models. **Unified Virtual Addressing (UVA)**: CUDA's UVA (since CUDA 4.0) maps CPU and GPU memory into a single virtual address space. Any pointer can be dereferenced by either CPU or GPU — the runtime determines the physical location and handles data migration. This eliminates the need for separate host/device pointer management. **CUDA Unified Memory**: Building on UVA, unified memory (managed memory) provides automatic page migration between CPU and GPU on demand. When the GPU accesses a page resident in CPU memory, a **page fault** triggers migration to GPU VRAM (and vice versa). The page fault mechanism (available since Pascal/sm_60) enables: **memory oversubscription** — GPU kernels can access more memory than physical VRAM by paging to system memory; **simplified programming** — no explicit cudaMemcpy calls; and **prefetch hints** — cudaMemPrefetchAsync allows applications to guide the migration system. **GPU Page Table Architecture**: Modern GPUs (NVIDIA Ampere and later) implement multi-level page tables similar to CPU MMUs. GPU page sizes are typically larger (64KB-2MB versus CPU's 4KB-2MB) to amortize TLB miss overhead and match GPU's coalesced access patterns. GPU TLBs are organized per-SM with L1 TLB and shared L2 TLB. TLB misses are expensive on GPUs because they stall thousands of threads simultaneously. **Memory Oversubscription**: When GPU VRAM is exhausted, pages are evicted to system memory. The GPU runtime implements a page replacement policy (LRU-based or access-frequency-based). Performance degrades as oversubscription increases because: PCIe/NVLink bandwidth (32-900 GB/s) is far below GPU memory bandwidth (~3 TB/s), and page faults stall warps until migration completes. However, oversubscription enables running workloads that previously required model sharding or data streaming. **Access Counters and Prefetching**: Hardware access counters track page access frequency and locality. The driver uses this telemetry for intelligent page placement: frequently-accessed pages migrate to VRAM, cold pages demote to system memory. Prefetching algorithms predict future access patterns (based on sequential detection or application hints) and migrate pages proactively. **Multi-GPU Memory Management**: In multi-GPU systems, page migration extends across GPUs. NVLink provides higher bandwidth for inter-GPU migration than PCIe. NVIDIA's multi-GPU memory management enables a single GPU kernel to transparently access memory on any GPU in the system, with the mapping and migration handled by the driver. **GPU virtual memory has transformed GPU programming from explicit, error-prone memory management to a more accessible model — enabling larger problems, simpler code, and transparent memory tiering across the heterogeneous memory hierarchy of modern computing systems.**