gpu memory management cuda

**GPU Memory Management** is **the systematic allocation, transfer, and optimization of data across CPU and GPU memory spaces to maximize performance and minimize overhead** — where understanding the trade-offs between pageable memory (convenient but slow), pinned memory (2-10× faster transfers), unified memory (automatic but overhead), and device memory (fastest but manual) enables developers to achieve 80-100% of theoretical memory bandwidth (1.5-3 TB/s on modern GPUs) through techniques like asynchronous transfers that overlap with computation, memory pooling that eliminates allocation overhead (5-50ms per allocation), and proper synchronization that avoids unnecessary CPU-GPU stalls, making memory management the critical factor in GPU application performance where poor memory management can reduce throughput by 5-10× through excessive transfers, synchronization overhead, and bandwidth underutilization. **Memory Types and Characteristics:** - **Device Memory**: GPU global memory; allocated with cudaMalloc(); 40-80GB capacity on modern GPUs; 1.5-3 TB/s bandwidth; fastest for GPU access; requires explicit CPU-GPU transfers - **Pinned (Page-Locked) Memory**: CPU memory locked in physical RAM; allocated with cudaMallocHost() or cudaHostAlloc(); 2-10× faster transfers than pageable; limited resource (system RAM); enables async transfers - **Pageable Memory**: standard CPU memory; malloc() or new; must be staged through pinned memory for GPU transfer; slower but unlimited; default for most allocations - **Unified Memory**: single address space for CPU and GPU; cudaMallocManaged(); automatic migration; convenient but 2-5× overhead vs explicit; good for prototyping - **Managed Memory**: subset of unified memory; automatic prefetching and eviction; cudaMemPrefetchAsync() for hints; 50-80% of explicit performance **Memory Allocation Strategies:** - **Pre-Allocation**: allocate all memory at initialization; reuse across iterations; eliminates allocation overhead (5-50ms per cudaMalloc); critical for performance - **Memory Pooling**: maintain pool of pre-allocated buffers; allocate from pool instead of cudaMalloc; 10-100× faster allocation; custom allocators or CUB device allocator - **Allocation Size**: large allocations (>1MB) more efficient; small allocations have high overhead; batch small allocations into single large allocation - **Alignment**: 256-byte alignment for optimal coalescing; cudaMalloc provides automatic alignment; manual alignment with __align__ for shared memory **Memory Transfer Optimization:** - **Asynchronous Transfers**: cudaMemcpyAsync() with pinned memory; overlaps with kernel execution; requires streams; 30-60% throughput improvement - **Batching**: combine multiple small transfers into single large transfer; reduces overhead; 2-5× faster for many small transfers - **Bidirectional Transfers**: overlap H2D and D2H transfers; use separate streams; 2× throughput vs sequential; requires 2 copy engines - **Zero-Copy**: access pinned host memory directly from GPU; cudaHostAlloc(cudaHostAllocMapped); avoids explicit transfer; slower than device memory but useful for infrequent access **Pinned Memory Best Practices:** - **Allocation**: cudaMallocHost() or cudaHostAlloc(); use for all data transferred to/from GPU; 2-10× faster than pageable - **Limitations**: limited by system RAM; excessive pinned memory reduces system performance; typical limit 50-80% of system RAM - **Portable Pinned**: cudaHostAllocPortable flag; accessible from all CUDA contexts; useful for multi-GPU; slight overhead - **Write-Combined**: cudaHostAllocWriteCombined; faster CPU writes, slower reads; use for data written by CPU, read by GPU **Unified Memory:** - **Automatic Migration**: pages migrate between CPU and GPU on demand; page faults trigger migration; 2-5× overhead vs explicit - **Prefetching**: cudaMemPrefetchAsync() prefetches to GPU; reduces page faults; 50-80% of explicit performance; good for prototyping - **Access Counters**: track which processor accesses data; optimizes placement; cudaMemAdvise() provides hints; 30-60% improvement - **Oversubscription**: allocate more than GPU memory; automatic eviction; enables large datasets; 2-10× slower than fitting in GPU memory - **When to Use**: rapid prototyping, irregular access patterns, CPU-GPU collaboration; production code prefers explicit for performance **Memory Synchronization:** - **cudaDeviceSynchronize()**: waits for all GPU operations; expensive (5-10ms); use sparingly; blocks CPU thread - **cudaStreamSynchronize()**: waits for specific stream; less expensive than device sync; 1-5ms; use for fine-grained control - **cudaEventSynchronize()**: waits for event; lightweight; <1ms; preferred for synchronization - **Implicit Sync**: cudaMemcpy() (non-async), cudaMalloc(), cudaFree() synchronize all streams; avoid in performance-critical code **Memory Bandwidth Optimization:** - **Coalesced Access**: threads in warp access consecutive addresses; 128-byte aligned; achieves 100% bandwidth; stride-1 optimal - **Vectorized Transfers**: use float4, int4 for 128-bit transfers; 2-4× fewer transactions; improves bandwidth utilization - **Measure Bandwidth**: achieved bandwidth / peak bandwidth; target 80-100%; Nsight Compute reports memory throughput - **Bottleneck Identification**: <50% bandwidth indicates access pattern problems; optimize coalescing, alignment, stride **Multi-GPU Memory Management:** - **Peer-to-Peer Access**: cudaDeviceEnablePeerAccess(); direct GPU-to-GPU memory access; requires NVLink or PCIe P2P; 5-10× faster than host staging - **Peer Copies**: cudaMemcpyPeer() or cudaMemcpyPeerAsync(); explicit GPU-to-GPU transfer; 900 GB/s with NVLink on A100; 64 GB/s with PCIe 4.0 - **Unified Memory Multi-GPU**: automatic migration between GPUs; convenient but overhead; explicit peer access preferred for performance - **Memory Affinity**: allocate memory on GPU where it's primarily used; reduces cross-GPU traffic; cudaSetDevice() before allocation **Memory Pooling Implementation:** - **CUB Device Allocator**: CUDA Unbound (CUB) library provides caching allocator; 10-100× faster than cudaMalloc; automatic memory reuse - **Custom Allocators**: implement application-specific pooling; pre-allocate large buffer; sub-allocate from buffer; eliminates cudaMalloc overhead - **PyTorch Caching**: PyTorch automatically pools GPU memory; torch.cuda.empty_cache() releases unused memory; generally efficient - **Memory Fragmentation**: pooling can cause fragmentation; periodic defragmentation or size-class pools mitigate; monitor with cudaMemGetInfo() **Memory Debugging:** - **cuda-memcheck**: detects out-of-bounds access, race conditions, uninitialized memory; run with cuda-memcheck ./app; 10-100× slowdown - **Compute Sanitizer**: newer tool replacing cuda-memcheck; more features; better performance; detects memory leaks - **cudaMemGetInfo()**: queries free and total memory; useful for monitoring; call periodically to detect leaks - **CUDA_LAUNCH_BLOCKING=1**: serializes operations; easier debugging; disables async; use only for debugging **Memory Profiling:** - **Nsight Systems**: timeline view; shows memory transfers; identifies transfer bottlenecks; visualizes CPU-GPU interaction - **Nsight Compute**: detailed memory metrics; bandwidth utilization, cache hit rates, coalescing efficiency; guides optimization - **nvprof**: deprecated but still useful; quick memory transfer overview; --print-gpu-trace shows all transfers - **Metrics**: transfer time, achieved bandwidth, transfer size, frequency; target 80-100% of peak bandwidth **Common Pitfalls:** - **Excessive Transfers**: transferring data every iteration; keep data on GPU when possible; 5-10× slowdown from unnecessary transfers - **Small Transfers**: many small transfers have high overhead; batch into larger transfers; 2-5× improvement - **Synchronous Transfers**: cudaMemcpy() blocks; use cudaMemcpyAsync() with pinned memory; 30-60% improvement - **Pageable Memory**: using malloc() for GPU transfers; 2-10× slower than pinned; always use cudaMallocHost() - **Memory Leaks**: forgetting cudaFree(); accumulates over time; monitor with cudaMemGetInfo(); use RAII wrappers **Advanced Techniques:** - **Mapped Memory**: CPU memory accessible from GPU; cudaHostAlloc(cudaHostAllocMapped); avoids explicit transfer; useful for infrequent access - **Texture Memory**: 2D/3D cached memory; cudaCreateTextureObject(); benefits spatial locality; 2-10× speedup for irregular access - **Constant Memory**: 64KB read-only cache; __constant__ qualifier; broadcast to all threads; 2-5× faster than global for uniform access - **Shared Memory**: on-chip SRAM; 164KB per SM on A100; 100× faster than global; explicit programmer control **Memory Hierarchy Strategy:** - **Hot Data**: frequently accessed; keep in device memory; never transfer; examples: model weights, intermediate activations - **Warm Data**: occasionally accessed; transfer once, reuse; examples: input batches, labels - **Cold Data**: rarely accessed; keep on CPU, transfer on demand; examples: validation data, checkpoints - **Streaming Data**: continuous flow; pipeline with async transfers; overlap with computation; examples: video frames, sensor data **Performance Targets:** - **Transfer Bandwidth**: 80-100% of peak (10-25 GB/s PCIe, 900 GB/s NVLink); use pinned memory and async transfers - **Allocation Overhead**: <1% of total time; use memory pooling; pre-allocate when possible - **Synchronization Overhead**: <5% of total time; minimize sync points; use async operations and streams - **Memory Utilization**: 70-90% of GPU memory; higher utilization improves efficiency; leave 10-30% for fragmentation and overhead **Best Practices:** - **Pre-Allocate**: allocate all memory at initialization; reuse across iterations; eliminates allocation overhead - **Pinned Memory**: use cudaMallocHost() for all CPU-GPU transfers; 2-10× faster than pageable - **Async Transfers**: use cudaMemcpyAsync() with streams; overlap with computation; 30-60% improvement - **Minimize Transfers**: keep data on GPU; transfer only when necessary; 5-10× improvement - **Profile**: use Nsight Systems to identify transfer bottlenecks; optimize based on data; measure achieved bandwidth GPU Memory Management is **the foundation of efficient GPU computing** — by understanding the trade-offs between memory types and applying techniques like pinned memory allocation, asynchronous transfers, and memory pooling, developers achieve 80-100% of theoretical bandwidth and eliminate allocation overhead, making proper memory management the difference between applications that achieve 10% or 90% of GPU potential where poor memory management can reduce throughput by 5-10× through excessive transfers and synchronization overhead.