gpu memory pool

**GPU Memory Pool Allocators** are the **caching memory management systems that maintain pre-allocated pools of GPU memory to eliminate the overhead of frequent cudaMalloc/cudaFree calls** — reducing allocation latency from milliseconds to microseconds, preventing memory fragmentation, and enabling the rapid tensor allocation/deallocation patterns required by deep learning frameworks. **The Problem with Raw CUDA Allocation** - `cudaMalloc()`: ~1-10 ms per call — extremely slow (requires GPU driver interaction, page table updates). - **Deep learning**: Each training iteration allocates/frees hundreds of tensors. - Without pooling: 200 allocations × 5 ms = 1 second of pure allocation overhead per iteration. - With pooling: 200 allocations × 5 μs = 1 ms — 1000x faster. **How Caching Allocators Work** 1. **First allocation**: Pool calls `cudaMalloc` for a **large block** (e.g., 2GB). 2. **User requests 256MB**: Pool carves out 256MB from the large block — returns pointer. 3. **User frees 256MB**: Pool marks the segment as available — does NOT call `cudaFree`. 4. **Next 256MB request**: Pool reuses the freed segment — zero allocation overhead. 5. **Pool grows**: If existing blocks are insufficient, allocate another large block. **PyTorch CUDA Caching Allocator** - Default allocator for all PyTorch GPU tensors. - Maintains separate pools for **small** (< 1MB) and **large** (≥ 1MB) allocations. - Uses **best-fit** strategy with block splitting to minimize fragmentation. - `torch.cuda.memory_summary()`: Shows allocated, reserved, and fragmented memory. - `torch.cuda.empty_cache()`: Returns unused cached blocks to CUDA (but doesn't help with fragmentation). **Memory Fragmentation** - Even with pooling, **fragmentation** occurs: Many small free blocks but no contiguous space for a large allocation. - Example: 8GB reserved, 2GB in use, but largest free block is only 500MB → cannot allocate 1GB tensor. - **Mitigation**: PyTorch 2.x uses `expandable_segments` configuration to reduce OS-level fragmentation. **CUDA Memory Pool API (CUDA 11.2+)** - `cudaMemPool_t`: Native CUDA memory pool support. - `cudaMallocAsync()` / `cudaFreeAsync()`: Stream-ordered allocation — allocation tied to CUDA stream. - Benefit: GPU hardware manages allocation ordering — further reduces synchronization overhead. **Memory Management Best Practices** - **Pre-allocate**: Allocate maximum-size tensors once at startup, reuse buffers. - **Gradient accumulation**: Process smaller micro-batches to reduce peak memory. - **Mixed precision**: FP16/BF16 tensors use half the memory of FP32. - **Activation checkpointing**: Trade compute for memory by recomputing activations during backward. GPU memory pool allocators are **essential infrastructure for all GPU computing frameworks** — without them, the rapid tensor allocation patterns of modern deep learning and scientific computing would be throttled by driver-level allocation overhead, making interactive and training workloads impractically slow.