GPU Memory Hierarchy is the multi-level, bandwidth-stratified storage system combining registers, caches, shared memory, and DRAM, with fundamentally different access latencies and throughputs that dominate GPU application performance.
GPU Memory Hierarchy Levels
- Registers (Per-Thread): ~256 bytes per thread (Ampere). 10 cycle latency, full bandwidth (every thread accesses concurrently). Precious resource (limited total capacity).
- L1 Cache (Per-SM): 32-128 KB per SM. 20-30 cycle latency, full bandwidth. Caches global memory loads if enabled. Per-SM coherence (no cross-SM coherence in L1).
- Shared Memory (Per-SM): 48-96 KB per SM, programmer-managed. 30 cycle latency, full bandwidth (if bank-conflict free). Explicit allocation in kernel parameters.
- L2 Cache (GPU-wide): 4-40 MB (varies by GPU). 100-200 cycle latency, shared across all SMs. Victim cache for L1, also caches uncached loads.
- HBM/GDDR (Main Memory): 16-80 GB on GPU. 200-500 cycle latency, peak bandwidth 2 TB/s (HBM2e A100) vs 700 GB/s (GDDR6x). Shared memory bus (all SMs contend).
Bandwidth Characteristics at Each Level
- Register Bandwidth: ~14-20 TB/s per SM (Ampere). All threads access simultaneously. Bottleneck: register count, not bandwidth.
- L1 Bandwidth: Limited by L1 port width. ~64 bytes per cycle typical (matching SM bus width). Sufficient for most kernels if L1 hits.
- L2 Bandwidth: Shared, measured as aggregate across all SMs. Peak = L2 frequency × port width. Typically 1-2 TB/s.
- DRAM Bandwidth: HBM2e 2 TB/s peak (Ampere A100). GDDR6X ~700 GB/s (RTX GPUs). Practical sustained: 80-90% of peak (protocol overhead, command latency).
Coalescing Rules for Global Memory
- Coalescing Requirement: 32 consecutive threads access 32 consecutive 4-byte words (128 bytes). Hardware merges into single 128-byte transaction.
- Coalescing Efficiency: Perfect coalescing = 1 transaction per 32 loads. Scattered access = 32 transactions (one per load). Cache size impacts coalescing benefit.
- Cache Benefits: If coalesced access pattern fits in L1/L2, subsequent accesses hit cache (no additional DRAM traffic). Cache reduces importance of perfect coalescing.
- Coalescing Patterns: Stride-1 (consecutive access) perfect. Stride-2 requires 2 transactions. Irregular access (indices from array) uses cache to recover.
Bank Conflict in Shared Memory
- Bank Architecture: 32 banks, one per thread (Ampere). Thread i accesses bank (i mod 32). 32-bit word = bank, 64-bit double = spans 2 banks.
- Conflict Condition: Multiple threads accessing same bank in same cycle. Results in serialization (32 way conflict worst case = 32x slowdown).
- Conflict Avoidance: Stride-1 access pattern (thread i accesses bank i) conflict-free. Stride-32 (all threads same bank) severe conflict. Padding arrays alleviates strides causing conflicts.
- Broadcast: Special case: all threads read same location (broadcast, no conflict). Hardware optimization reduces to single access.
L2 Cache Policies and Control
- Cache Mode: Persistent (caching) or streaming (bypass). Persistent mode caches data expected to be reused. Streaming bypasses cache (saves cache space).
- Persistent Mode: Data cached in L2, reused. Beneficial for loops, stencil operations with repeated access.
- Streaming Mode: Each load bypasses L2. Useful for one-time accesses (reduce cache pollution, prioritize cache space for other kernels).
- Coherency: L2 cache hardware coherent (all SM L1 coherence via L2). Shared memory coherence SW responsibility (barriers, atomics).
Unified Memory and Page Migration
- Unified Memory Abstraction: Single virtual address space for CPU and GPU. malloc() returns GPU-accessible pointer. Implicit data migration (CPU ↔ GPU) as needed.
- Page Fault Mechanism: Page faults detect out-of-locality access. OS migrates page on fault (100-1000µs latency). Transparent but potentially slow.
- Prefetch Optimization: cudaMemPrefetchAsync() explicitly migrate pages to GPU before kernel execution. Avoids page-fault latency.
- Managed Memory Overhead: Page table management overhead ~5-15%. For frequently-migrating pages, explicit cudaMemcpy faster.
Prefetching Strategies
- Hardware Prefetching: GPU hardware prefetches next-line (adjacent cache line) on load miss. Reduces miss latency for streaming access (stride-1).
- Software Prefetching: Explicitly load data ahead of use. ldg() intrinsic performs load-to-cache (not register). Allows computation to overlap with pending loads.
- Double Buffering: Prefetch next iteration's data while current iteration computes. Hides DRAM latency via pipelining.
- Stream Prefetching: For streaming access patterns, hardware prefetch usually sufficient. For irregular patterns, software prefetch + synchronization necessary.
Memory Access Optimization Case Studies
- Matrix Multiplication (GEMM): Transposed B for coalescing (column-major access patterns). Tiled computation (shared memory) reduces DRAM bandwidth 10x.
- Stencil Computation: Halo exchange via global memory (coalescing important). Shared memory staging reduces DRAM by 4-10x for interior points.
- Sparse Matrix-Vector Product: Irregular access patterns. Reordering rows improves coalescing. Compression (CSR) reduces data footprint.