GPU Memory Management — understanding the GPU memory hierarchy and managing data transfers between host (CPU) and device (GPU) memory to avoid bottlenecks that dominate application performance.

Memory Spaces in CUDA

• Global memory: Main GPU DRAM (HBM or GDDR). Large (16–80GB), high bandwidth (1–3 TB/s), but high latency (~400 cycles)
• Shared memory: On-chip SRAM per SM. Small (48–228KB), very fast (~30 cycles). Programmer-managed cache
• Registers: Per-thread. Fastest. Limited (~255 per thread)
• Constant memory: Read-only, cached. Good for broadcast data
• Texture memory: Read-only with spatial caching. Good for 2D access patterns

Host-Device Transfers

cudaMalloc(&d_ptr, size);      // Allocate device memory
cudaMemcpy(d_ptr, h_ptr, size, cudaMemcpyHostToDevice);  // Upload
kernel<<<grid, block>>>(d_ptr);  // Compute
cudaMemcpy(h_ptr, d_ptr, size, cudaMemcpyDeviceToHost);  // Download

• PCIe bandwidth: ~25 GB/s (PCIe 4.0 x16). GPU memory bandwidth: ~2000 GB/s → 80x difference
• Minimize transfers! Overlap compute with transfers using CUDA streams

Unified Memory

• cudaMallocManaged() — single pointer accessible from CPU and GPU
• Hardware page migration between CPU and GPU on demand
• Simpler programming but can have performance overhead from page faults

Memory management is the single most important performance factor in GPU programming — compute is rarely the bottleneck, memory is.