gpu shared memory optimization

**GPU Shared Memory Optimization** is the **performance-critical programming technique that uses the GPU's fast, software-managed on-chip memory (shared memory / scratchpad) to cache frequently-accessed data, enabling data reuse across threads within a thread block while avoiding repeated expensive global memory accesses — where proper use can improve kernel performance by 5-20x but improper use (bank conflicts, insufficient occupancy) can negate the benefits entirely**. **Shared Memory Architecture** Shared memory is a low-latency (~5 cycles), high-bandwidth on-chip SRAM organized into 32 banks (each 4 bytes wide). All threads in a thread block share the same shared memory instance (configurable 48-164 KB per SM on modern GPUs). Access latency is ~100x lower than global memory (HBM: ~500 cycles). **Bank Conflicts** The 32 banks can each serve one 4-byte access per cycle simultaneously. If two or more threads in the same warp access different addresses in the same bank, the accesses are serialized (N-way bank conflict → N cycles). Conflict-free access patterns: - **Linear stride-1**: Thread i accesses word i → each thread hits a different bank. No conflict. - **Linear stride-2**: Threads 0,16 both hit bank 0; threads 1,17 both hit bank 1 → 2-way conflict everywhere. - **Stride-32**: All threads hit the same bank → 32-way conflict (worst case). Solution: Pad the shared memory array to offset the stride. For a 32 × 32 float array, declaring it as float tile[32][33] shifts each row by one bank, eliminating conflicts for column access. **Common Optimization Patterns** - **Tiling (Matrix Multiply)**: Load a tile of matrix A and matrix B into shared memory. Each thread in the block reuses these tiles for multiple multiply-accumulate operations. For a 32×32 tile, each global memory load is reused 32 times, reducing global memory traffic by 32x. - **Stencil Computation**: Load a tile plus halo (boundary elements needed by border threads) into shared memory. Compute the stencil entirely from shared memory. Avoids redundant global memory reads of overlapping halo regions. - **Histogram / Reduction**: Accumulate partial results in shared memory across the thread block, then write a single consolidated result to global memory. **Configuration Tradeoffs** - **Shared Memory vs. L1 Cache**: Modern GPUs allow configuring the partition between shared memory and L1 cache (e.g., prefer 48KB shared + 112KB L1, or 164KB shared + 0KB L1 on H100). Kernels with explicit tiling benefit from more shared memory; kernels with irregular access patterns benefit from more L1. - **Occupancy Impact**: More shared memory per block means fewer blocks can run concurrently per SM. If a kernel uses 48KB shared/block and the SM has 164KB total, only 3 blocks run simultaneously. Reducing shared memory usage to 32KB allows 5 blocks → higher occupancy and better latency hiding. GPU Shared Memory is **the parallel programmer's most powerful tool for bridging the bandwidth gap between compute and memory** — a manually-managed cache that, when used correctly, transforms memory-bound kernels into compute-bound kernels.