cpu cache optimization
**CPU Cache Optimization** is the **systematic restructuring of data layouts and access patterns to maximize cache hit rates and minimize data movement through the memory hierarchy**, exploiting the principle of locality — the tendency of programs to access data that is nearby in time (temporal) or space (spatial) to recently accessed data — to bridge the 100x latency gap between L1 cache and main memory.
Modern CPUs spend more transistors on cache than on computation. A cache miss to DRAM costs 200-300 cycles — equivalent to 200-300 wasted ALU operations. For memory-bound workloads, cache optimization often delivers larger speedups than algorithmic improvements.
**Cache Hierarchy**:
| Level | Size | Latency | Bandwidth | Line Size |
|-------|------|---------|-----------|----------|
| **L1 data** | 32-96 KB per core | 4-5 cycles | ~2-4 TB/s | 64 bytes |
| **L2** | 256 KB-2 MB per core | 10-14 cycles | ~1-2 TB/s | 64 bytes |
| **L3** | 8-256 MB shared | 30-50 cycles | ~400-800 GB/s | 64 bytes |
| **DRAM** | 16-512 GB | 200-300 cycles | ~50-200 GB/s | 64 bytes |
**Loop Tiling (Blocking)**: The most impactful optimization for nested loops over large arrays. Instead of processing entire rows/columns (which exceed cache size), process small tiles that fit entirely in L1 or L2 cache. For matrix multiplication: tile sizes of 32-64 elements (fitting in L1) achieve 3-5x speedup over untiled code because each element is reused multiple times from cache rather than refetched from DRAM.
**Data Layout Optimization**: **Array of Structures (AoS) vs. Structure of Arrays (SoA)**: AoS (`struct {x,y,z,w} particles[N]`) packs all fields per element but wastes bandwidth when only one field is accessed. SoA (`float x[N], y[N], z[N], w[N]`) enables accessing one field with perfect spatial locality and enables vectorization. **Padding**: add padding to avoid false sharing (two threads on different cores accessing different variables that share a cache line, causing coherence traffic) and to avoid cache set conflicts.
**Prefetching**: CPUs have hardware prefetchers that detect sequential and strided access patterns and load cache lines before they're needed. When hardware prefetching fails (irregular access, pointer chasing), software prefetch hints (`__builtin_prefetch()`, `_mm_prefetch()`) tell the CPU to begin loading data N iterations ahead. The prefetch distance must balance: too short and data arrives late (cache miss), too long and data is evicted before use.
**Cache-Oblivious Algorithms**: Achieve near-optimal cache performance at all levels of the hierarchy without knowing cache sizes. The key technique: recursive divide-and-conquer until base cases fit in the smallest cache. The van Emde Boas memory layout for trees reorders nodes to match cache line boundaries. Funnel sort arranges merge operations to automatically match any cache size.
**Measurement**: Hardware performance counters (perf stat, VTune, LIKWID) measure L1/L2/L3 miss rates, providing direct feedback: >5% L1 miss rate typically indicates optimization opportunity; L3 miss rates correlate directly with DRAM bandwidth consumption. Cache-aware optimization targets the dominant miss source.
**CPU cache optimization is the performance engineering discipline that respects the physical reality of memory — computation is essentially free compared to data movement, and the programmer who structures code to keep data in cache achieves performance that the programmer who ignores the cache hierarchy will never match.**