Virtual Memory
Virtual memory is the hardware-software mechanism that gives every process the illusion of a private, contiguous address space spanning terabytes — even though physical RAM is shared, fragmented, and far smaller — by translating virtual addresses to physical addresses through page tables managed by the OS and enforced by the MMU (memory management unit) in hardware. Virtual memory enables process isolation (one process cannot access another's data), overcommitment (allocate more memory than physically exists), memory-mapped files, and demand paging (load only what's needed). Every modern processor — from phone SoCs to AI servers — implements virtual memory.
How it works — address translation. When a program accesses address 0x7FFF_1234_5678, the MMU splits it into a page number + offset, walks the multi-level page table (4 or 5 levels on x86-64), and produces the physical address. If the page is not in RAM (page fault), the OS loads it from storage (swap or file). This happens on every memory access — so it must be fast:
The Translation Lookaside Buffer (TLB) caches recent translations so >99% of accesses resolve in 1 cycle without walking the page table.
Page table structure (x86-64, 4-level):
| Level | Bits used | Entries | Each entry covers | Name |
|---|---|---|---|---|
| PML4 (L4) | [47:39] | 512 | 512 GB | Page map level 4 |
| PDPT (L3) | [38:30] | 512 | 1 GB | Page directory pointer |
| PD (L2) | [29:21] | 512 | 2 MB | Page directory |
| PT (L1) | [20:12] | 512 | 4 KB | Page table |
| Offset | [11:0] | — | 1 byte (within 4 KB page) | — |
Total addressable: 2⁴⁸ bytes = 256 TB of virtual space per process (5-level paging extends to 2⁵⁷ = 128 PB).
TLB — the critical performance structure. A full page-table walk takes 4 memory accesses (one per level) × ~5 ns each = 20 ns penalty. The TLB caches 64–4096 recent translations in a small, fast associative array:
| TLB type | Entries | Page size | Hit rate | Miss penalty |
|---|---|---|---|---|
| L1 ITLB | 64–128 | 4 KB | >99% | ~7 cycles (L2 TLB) |
| L1 DTLB | 64–96 | 4 KB | >99% | ~7 cycles |
| L2 STLB (unified) | 1024–4096 | 4 KB / 2 MB | >99.9% | ~20 ns (page walk) |
| Huge-page TLB | 32–64 | 2 MB or 1 GB | ~100% (AI workloads) | — |
Huge pages — why AI workloads need them. A 80 GB HBM allocation at 4 KB pages requires 20 million page-table entries and 20 million TLB slots (impossible — TLBs are small). At 2 MB huge pages: only ~40,000 entries. At 1 GB pages: only 80 entries — fits entirely in the TLB with zero misses. This is why AI frameworks (PyTorch, JAX) and GPU drivers use huge pages for pinned memory: eliminating TLB misses on the massive tensor allocations that dominate AI workloads.
Virtual memory for GPU and AI chips:
- CPU MMU: full virtual memory — page tables in DRAM, hardware page-table walker, 4-level or 5-level, demand paging, copy-on-write.
- GPU (NVIDIA): separate GPU page tables managed by the driver. Unified Virtual Memory (UVM) allows CPU and GPU to share a virtual address space — the driver migrates pages between CPU DRAM and GPU HBM on demand (but at high cost; explicit placement is preferred for performance).
- AI ASIC (TPU): minimal or no virtual memory — DMA addresses are physical or use a simple IOMMU. The software runtime manages all allocation explicitly.
Page faults and demand paging. When a virtual page has no physical mapping (first access, swapped out, or memory-mapped file), a page fault triggers:
1. CPU traps to the OS kernel
2. OS determines the cause (valid fault → allocate page; invalid → segfault/kill)
3. OS finds a free physical frame (or evicts one to swap)
4. OS fills the page (zeroed, from file, from swap)
5. OS updates the page table, returns to user — process resumes
A page fault takes ~1–10 microseconds — acceptable for rare events, catastrophic if frequent (thrashing). AI training avoids this by pinning all memory upfront.
Virtual memory and the CFS platform. Virtual memory is the OS abstraction that lets multiple processes share hardware safely. The CFS operating-system keyword covers the broader OS context; the memory-hierarchy keyword covers the physical storage layers that VM maps onto; and the cache-coherence keyword covers how multiple cores agree on the contents of shared physical addresses that VM translates to.