cache memory
**Cache memory** is a small, fast storage hierarchy placed near compute units to keep recently used instructions and data from repeatedly traveling to slower, more energy-intensive memory. It works because programs have locality: data used now is likely to be used again soon, and data near a recently used address is likely to be used next. A cache does not increase the capacity of main memory. It changes the average cost of reaching it, turning a mix of one-cycle hits and hundred-cycle misses into the latency the processor actually experiences.
**A cache stores fixed-size blocks.** Main memory is divided into cache lines, commonly 64 bytes in CPUs. When a core requests one byte, the hierarchy usually transfers the entire containing line. Each resident line carries data plus metadata: a tag identifies the memory block, valid and dirty bits describe its state, replacement state helps choose a victim, and coherence state coordinates copies held by other cores. The address is interpreted as offset, set index, and tag. The offset selects a byte within the line, the index selects a set, and parallel tag comparison determines whether one of that set’s ways is a hit.
**The hierarchy trades speed for reach.** L1 caches are tiny and physically close enough to serve the pipeline in a few cycles. L2 is larger and slower. A shared last-level cache absorbs traffic from many cores and reduces accesses to DRAM. Translation lookaside buffers cache virtual-to-physical address translations, while instruction caches, data caches, micro-op caches, victim caches, write buffers, and prefetch buffers target different bottlenecks. GPUs add software-managed shared memory or scratchpad alongside hardware-managed caches because predictable tiled workloads benefit from explicit placement.
| Level / structure | Typical capacity | Approximate hit latency | Typical line / unit | Primary job |
|---|---:|---:|---:|---|
| CPU L1 data | 32–64 KiB per core | 3–5 cycles | 64 B line | Feed loads and stores every cycle |
| CPU L2 | 256 KiB–2 MiB per core | 10–20 cycles | 64 B line | Catch private working-set misses |
| Shared L3 / LLC | 8–128 MiB per socket | 30–70 cycles | 64 B line | Reduce DRAM and cross-core traffic |
| GPU L1 / shared-memory complex | 64–256 KiB per SM | tens of cycles | sectors / cache lines | Reuse tiles, textures, and local data |
| HBM or DDR main memory | GiB to TiB | hundreds of cycles | burst transfer | Hold the full active dataset |
Values vary by architecture, frequency, banking, contention, and whether the requested line is local to the relevant cache slice. The meaningful number is not a marketing latency in isolation; it is the delivered latency and bandwidth under the workload’s access pattern.
```svg
```
**Average memory access time makes the trade explicit.** For one cache level with hit time $T_h$, miss rate $m$, and miss penalty $P$, the average is
$$T_{avg} = T_h + m \cdot P$$
A one-percentage-point miss-rate reduction can matter more than shaving a fraction of a cycle from a hit when the miss penalty reaches DRAM. Multi-level analysis expands the penalty into the probability-weighted latency of L2, L3, memory, and coherence actions. Performance counters therefore need context: a high L1 miss rate may be harmless if L2 hits are fast, while a modest last-level miss rate can saturate memory bandwidth.
**Associativity controls placement freedom.** A direct-mapped cache gives each memory block one possible slot, making lookup fast but allowing unrelated addresses to collide. An N-way set-associative cache offers N candidate lines per set, reducing conflict misses at the cost of more tag comparisons, muxing, energy, and replacement state. A fully associative structure permits any location but scales poorly for large capacities. Designers choose associativity separately for data, instructions, translations, and specialized structures based on timing and conflict behavior.
**Misses have different causes.** Compulsory misses occur on the first reference to a line. Capacity misses occur because the active working set exceeds available storage. Conflict misses occur because mapping forces reusable lines into the same set. Coherence misses occur when another agent invalidates or changes a shared line. Translation misses delay address generation even if the data itself is cached. Classifying misses matters because a larger cache will not fix poor spatial locality, and higher associativity will not fix a streaming workload whose data is never reused.
**Replacement predicts future value from past behavior.** True least-recently-used replacement is expensive at high associativity, so real designs use pseudo-LRU trees, recency bits, re-reference predictors, insertion policies, or sampled adaptive schemes. Streaming data should often enter with low priority so it does not evict a hot working set. Prefetched lines may deserve different treatment from demand-fetched lines. The policy is part of the architecture: two caches with identical size and associativity can produce materially different performance.
**Writes add policy choices.** Write-through immediately propagates updates to the next level, simplifying some consistency behavior but increasing traffic. Write-back marks a line dirty and delays the lower-level update until eviction, saving bandwidth but requiring dirty state and a potentially expensive writeback path. Write-allocate fetches a missed line before updating it; no-write-allocate sends the write around the cache. Store buffers decouple retirement from cache access, combine adjacent writes, and forward recent values to dependent loads.
**Coherence makes private caches behave like shared memory.** When multiple cores cache the same physical line, a protocol such as MESI tracks whether each copy is modified, exclusive, shared, or invalid. A write generally obtains ownership and invalidates other copies. Directory structures avoid broadcasting every request across large systems, but add lookup and network traffic. False sharing occurs when independent variables occupy one line: cores repeatedly invalidate the line even though they do not logically share data. Padding or data-layout changes can outperform any hardware tuning in that case.
**SRAM physics sets the area and energy floor.** Most on-chip caches use six-transistor SRAM bitcells surrounded by wordline drivers, bitline precharge, sense amplifiers, decoders, redundancy, and error correction. Capacity does not scale as pure bitcell area because tags, peripheral circuits, banking, routing, and timing margins consume substantial space. Lower voltage saves dynamic energy but reduces read stability and write margin. Large caches are divided into banks and slices so only part of the array activates per access and wires remain short enough to meet timing.
**Banking creates bandwidth and conflicts.** Multiple banks allow independent accesses in parallel when addresses distribute well. If many requests map to one bank, they serialize even when total capacity is idle. GPUs expose this effect directly in shared memory, where thread addresses that hit distinct banks proceed together and bank conflicts require multiple transactions. CPU caches hide more of the scheduling, but bank selection, port count, load-store queues, and miss-status holding registers still determine delivered throughput.
**Prefetching spends bandwidth to buy time.** A next-line prefetcher captures sequential streams, stride predictors learn regular gaps, and more advanced engines correlate address histories. Useful prefetches arrive before demand and turn misses into hits. Late prefetches do not hide latency; inaccurate prefetches waste bandwidth, occupy cache lines, and can evict useful data. Confidence control and throttling are essential when many cores share a memory interface.
**Software controls locality through layout and tiling.** Loop interchange, blocking, structure-of-arrays layouts, compact objects, page placement, and NUMA affinity can change cache behavior without changing an algorithm’s mathematical result. Matrix multiplication is fast when tiles of the operands remain in cache or scratchpad long enough for many multiply-accumulate operations. Pointer-heavy graphs are difficult because future addresses are data-dependent. Measuring reuse distance and working-set size is more actionable than labeling an application simply “memory bound.”
**Reliability is built into the hierarchy.** Parity commonly protects tags and control state; error-correcting codes protect data arrays and larger lower-level caches. Scrubbing repairs correctable errors before another bit flips. Physical address hashing spreads adjacent lines across banks and reduces systematic hotspots. Inclusive and non-inclusive policies affect recovery and coherence. Because cache state is usually reconstructible from lower memory, some faults can invalidate a line and retry, but dirty data requires stronger protection.
**Cache design is a system optimization.** More capacity may reduce misses while increasing hit latency, area, leakage, and wire delay. Wider lines exploit spatial locality but waste bandwidth on sparse accesses. More ports raise throughput but make arrays larger. An inclusive shared cache simplifies snooping but duplicates private data. Chiplet systems add another question: whether remote cache capacity is worth fabric latency. Architects evaluate traces, simulators, queueing behavior, power models, and physical-design estimates together because no single cache metric predicts application performance.
**Use the hierarchy as a traffic filter.** Ask what data is reused, how soon it is reused, which agents share it, and what happens when it misses. Then place capacity and policy at the level that eliminates the most expensive traffic without slowing the common hit. That framing connects cache memory to CFS tools for SRAM, interconnect, parallelism, inference, and power: the cache is where program behavior meets circuits, layout, and the memory system.