Cache Memory

Cache memory is a small, very fast memory placed between a processor and the larger, slower memory behind it, holding copies of the data and instructions most likely to be needed again so that most accesses are answered quickly without a trip to main memory. It is transparent: programs do not name the cache or load it explicitly, because the hardware decides automatically what to keep, and a program that runs faster after being run once is usually benefiting from it. The word cache, from the French for a hiding place, is now used for any such stash of frequently used data, from browsers to databases, but the original and most important meaning is the hardware one.

Caches work because programs are not random, a regularity called locality of reference. Temporal locality means that something used recently is likely to be used again soon, such as a loop counter or a variable in a hot function; spatial locality means that data near something just used is likely to be used next, such as the next element of an array. Maurice Wilkes proposed the idea in 1965 as slave memory, and the IBM System/360 Model 85 in 1968 was the first commercial machine to include a cache. Intel's 80486 in 1989 put 8 KB of cache on the processor die itself, and today every serious processor has one. The reason caches are needed at all is a gap that has grown for decades: a processor can execute several instructions in a nanosecond, while a round trip to dynamic RAM takes on the order of tens of nanoseconds or more.

Caches are organized in levels and in fixed-size blocks called cache lines. A small, fastest level 1 cache, often tens of kilobytes per core and frequently split into instruction and data halves, sits closest to each core; a larger level 2 cache of hundreds of kilobytes to a few megabytes follows, and a level 3 cache of many megabytes is typically shared among all the cores on the chip. Data moves in lines, commonly 64 bytes on mainstream x86 and many ARM processors, so touching one byte pulls in its 63 neighbors. A request that finds its data in the cache is a hit; one that does not is a miss and goes to the next level, and the average access time depends heavily on the hit rate. Since the cache is small, each new line must displace another, and the hardware chooses what to evict with approximations of the least recently used policy. The register entry covers the storage inside the core that sits above the caches, and random access memory the main memory below them.

With several cores, caches must also agree with one another, and that coordination has a cost. Each core has its own private caches, so when one core writes to a line that another has a copy of, the copies must be invalidated or updated, which is the job of a cache coherence protocol such as MESI. This is invisible to a correct program but not to its speed: two threads updating different variables that happen to share one line repeatedly steal it from each other, a slowdown known as false sharing. Cache behavior is also a security matter, since a program can time memory accesses to learn which lines another has touched; the Spectre and Meltdown disclosures in January 2018 turned that timing channel into a headline vulnerability affecting nearly every modern processor.

The practical consequence is that the same algorithm can run many times faster or slower depending on how it walks through memory. Arrays are stored in rows in languages like C and in NumPy's default layout, so summing a large matrix row by row reads consecutive addresses and hits the cache, while summing it column by column strides across lines and can run dramatically slower despite identical arithmetic. Data scientists see the same effect when choosing a contiguous array over a list of Python objects, and when libraries use tiling, splitting a large matrix product into blocks that fit in cache. Common pitfalls are assuming that a bigger cache always helps, whereas it benefits only working sets that fit; assuming that cache is a part of the storage drives; and confusing the hardware cache with a web or application cache. AMD's 3D V-Cache, first sold in 2022, stacks extra cache vertically on the chip, an acknowledgment that for many workloads memory delay is the real limit.

Read cache memory as the short list of things within arm's reach: it speeds up a machine not by being large or smart, but by betting, correctly most of the time, that what was needed a moment ago and what sits next to it will be needed again.