cache hierarchy
A cache hierarchy is a layered arrangement of multiple cache memory levels between a processor and main memory, each successive level trading some speed for greater capacity, keeping frequently used data close to the processor while still supporting a much larger overall memory footprint.
```flowchart
{
"rows": [
{ "type": "nodes", "items": [
{ "title": "Main memory is large but too slow for every access", "sub": "processor would spend most of its time waiting on memory", "tone": "red" }
]},
{ "type": "arrow" },
{ "type": "group", "title": "Multiple cache levels layered between processor and memory", "items": [
{ "title": "Each level trades some speed for greater capacity", "sub": "frequently used data kept close to the processor", "tone": "blue" }
]},
{ "type": "arrow" },
{ "type": "nodes", "items": [
{ "title": "Processor gets fast access to most data it needs", "sub": "large overall memory footprint still supported", "tone": "green" }
]}
]
}
```
**Cache hierarchies exist because there's a fundamental tradeoff between memory speed and memory capacity, no single memory technology can be both extremely fast and extremely large at an affordable cost, so processors instead use several different levels together.** Since a small amount of extremely fast memory sitting closest to the processor can hold only a limited amount of data, while a much larger, more affordable memory sits further away and runs meaningfully slower, a cache hierarchy layers several such levels together, each trading some access speed for significantly more capacity than the level before it, so that frequently accessed data typically stays available at a fast level while the system as a whole still supports a much larger total memory footprint.
```svg
```
```svg
```
| Aspect | Higher cache level (e.g., L1) | Lower cache level / main memory |
|---|---|---|
| Access speed | Fastest | Progressively slower |
| Capacity | Smallest | Progressively larger |
| Distance from processor | Closest | Progressively farther |
| Common use | Most frequently accessed data | Broader, less frequently accessed data |
**Cache hierarchies rely on the principle of locality, the observation that programs tend to repeatedly access the same or nearby data over short periods, which is what makes keeping small amounts of hot data in fast caches so effective.** Because real programs tend to reuse the same data repeatedly and access data located near other recently used data, a phenomenon called locality, a relatively small fast cache holding just the currently relevant data can satisfy the large majority of memory requests, which is the core reason cache hierarchies work as well as they do in practice.
**A cache miss, when requested data isn't found in a given cache level, requires checking the next level down, and a full hierarchy of misses cascading all the way to main memory carries a significant performance penalty.** Because each cache level can only hold a subset of all data, a request for data not present in a given level, called a cache miss, requires checking the next level down, and in the worst case falling all the way through to main memory, which is dramatically slower than any cache level, making minimizing cache misses a central concern in processor performance.
**Modern processors typically implement at least three cache levels, commonly labeled L1 through L3, with L1 caches often further split into separate instruction and data caches for even faster specialized access.** Because instructions and data have somewhat different access patterns, many processors split their fastest, smallest L1 cache into two separate instruction and data caches, while larger, shared L2 and L3 caches further down the hierarchy typically serve both instructions and data together.
Read the cache hierarchy through a desk-shelf-warehouse lens: papers you're using right now sit on your desk for instant access, less urgent ones go on a nearby shelf that takes a bit longer to reach, and everything else waits in a distant warehouse — each step trades a little speed for a lot more storage room.