gpu memory hierarchy

**GPU memory hierarchy** is the **layered organization of storage levels with different capacities and latency-bandwidth characteristics** - effective kernel design depends on maximizing reuse in faster tiers and minimizing expensive global memory access. **What Is GPU memory hierarchy?** - **Definition**: Hierarchy from registers and on-chip caches to shared memory, L2 cache, and off-chip HBM. - **Speed Gradient**: Closer memories are smaller but faster, while larger memories are slower and higher latency. - **DL Relevance**: Memory movement often limits performance more than raw compute throughput. - **Optimization Principle**: Increase arithmetic intensity by reusing data before evicting to slower tiers. **Why GPU memory hierarchy Matters** - **Kernel Efficiency**: Poor hierarchy use leads to bandwidth stalls and low tensor-core utilization. - **Throughput Scaling**: Memory-aware kernels sustain higher effective FLOPs at large problem sizes. - **Energy Cost**: Reducing off-chip transfers lowers power consumption and thermal pressure. - **Model Performance**: Attention and activation-heavy workloads are especially memory hierarchy sensitive. - **Hardware ROI**: Understanding hierarchy is essential to realize the performance promised by modern GPUs. **How It Is Used in Practice** - **Access Pattern Design**: Use coalesced loads and tile reuse to maximize on-chip residency. - **Fusion Strategies**: Fuse adjacent operators to reduce intermediate writes to global memory. - **Profiler Guidance**: Track memory throughput and cache hit metrics to target bottleneck tiers. GPU memory hierarchy is **the dominant performance constraint in many deep learning kernels** - compute speed is unlocked only when data movement is engineered with hierarchy awareness.