GPU Shared Memory Optimization is the performance-critical programming technique that uses the GPU's fast, software-managed on-chip memory (shared memory / scratchpad) to cache frequently-accessed data, enabling data reuse across threads within a thread block while avoiding repeated expensive global memory accesses — where proper use can improve kernel performance by 5-20x but improper use (bank conflicts, insufficient occupancy) can negate the benefits entirely.

Shared Memory Architecture

Shared memory is a low-latency (~5 cycles), high-bandwidth on-chip SRAM organized into 32 banks (each 4 bytes wide). All threads in a thread block share the same shared memory instance (configurable 48-164 KB per SM on modern GPUs). Access latency is ~100x lower than global memory (HBM: ~500 cycles).

Bank Conflicts

The 32 banks can each serve one 4-byte access per cycle simultaneously. If two or more threads in the same warp access different addresses in the same bank, the accesses are serialized (N-way bank conflict → N cycles). Conflict-free access patterns:

• Linear stride-1: Thread i accesses word i → each thread hits a different bank. No conflict.
• Linear stride-2: Threads 0,16 both hit bank 0; threads 1,17 both hit bank 1 → 2-way conflict everywhere.
• Stride-32: All threads hit the same bank → 32-way conflict (worst case).

Solution: Pad the shared memory array to offset the stride. For a 32 × 32 float array, declaring it as float tile[32][33] shifts each row by one bank, eliminating conflicts for column access.

Common Optimization Patterns

• Tiling (Matrix Multiply): Load a tile of matrix A and matrix B into shared memory. Each thread in the block reuses these tiles for multiple multiply-accumulate operations. For a 32×32 tile, each global memory load is reused 32 times, reducing global memory traffic by 32x.
• Stencil Computation: Load a tile plus halo (boundary elements needed by border threads) into shared memory. Compute the stencil entirely from shared memory. Avoids redundant global memory reads of overlapping halo regions.
• Histogram / Reduction: Accumulate partial results in shared memory across the thread block, then write a single consolidated result to global memory.

Configuration Tradeoffs

• Shared Memory vs. L1 Cache: Modern GPUs allow configuring the partition between shared memory and L1 cache (e.g., prefer 48KB shared + 112KB L1, or 164KB shared + 0KB L1 on H100). Kernels with explicit tiling benefit from more shared memory; kernels with irregular access patterns benefit from more L1.
• Occupancy Impact: More shared memory per block means fewer blocks can run concurrently per SM. If a kernel uses 48KB shared/block and the SM has 164KB total, only 3 blocks run simultaneously. Reducing shared memory usage to 32KB allows 5 blocks → higher occupancy and better latency hiding.

GPU Shared Memory is the parallel programmer's most powerful tool for bridging the bandwidth gap between compute and memory — a manually-managed cache that, when used correctly, transforms memory-bound kernels into compute-bound kernels.