gpu memory management
**GPU Memory Management** — understanding the GPU memory hierarchy and managing data transfers between host (CPU) and device (GPU) memory to avoid bottlenecks that dominate application performance.
**Memory Spaces in CUDA**
- **Global memory**: Main GPU DRAM (HBM or GDDR). Large (16–80GB), high bandwidth (1–3 TB/s), but high latency (~400 cycles)
- **Shared memory**: On-chip SRAM per SM. Small (48–228KB), very fast (~30 cycles). Programmer-managed cache
- **Registers**: Per-thread. Fastest. Limited (~255 per thread)
- **Constant memory**: Read-only, cached. Good for broadcast data
- **Texture memory**: Read-only with spatial caching. Good for 2D access patterns
**Host-Device Transfers**
```
cudaMalloc(&d_ptr, size); // Allocate device memory
cudaMemcpy(d_ptr, h_ptr, size, cudaMemcpyHostToDevice); // Upload
kernel<<>>(d_ptr); // Compute
cudaMemcpy(h_ptr, d_ptr, size, cudaMemcpyDeviceToHost); // Download
```
- PCIe bandwidth: ~25 GB/s (PCIe 4.0 x16). GPU memory bandwidth: ~2000 GB/s → 80x difference
- Minimize transfers! Overlap compute with transfers using CUDA streams
**Unified Memory**
- `cudaMallocManaged()` — single pointer accessible from CPU and GPU
- Hardware page migration between CPU and GPU on demand
- Simpler programming but can have performance overhead from page faults
**Memory management** is the single most important performance factor in GPU programming — compute is rarely the bottleneck, memory is.