Lpddr6 & Kv Cache

LPDDR6 and the KV cache are two halves of the same problem: large-language-model inference is limited by how fast memory can be read, not by how fast a chip can multiply. LPDDR6 roughly doubles per-pin bandwidth over LPDDR5X while improving energy per bit, which matters because the KV cache — the attention state a transformer keeps for every token it has already seen — must be streamed out of memory in full for every single new token generated. Processing-in-memory (PIM) attacks the same wall from the opposite direction by performing arithmetic inside the DRAM so the data never has to move at all.

What the KV cache is and why it dominates. During autoregressive generation a transformer caches the key and value projections for every token processed so far, so it does not recompute them. The cache size grows linearly with context length:

$$ Bytes_{KV} = 2 \times L \times H \times d_{head} \times N_{tokens} \times P $$

where $L$ is layers, $H$ is key/value heads, $d_{head}$ the head dimension, $N_{tokens}$ the context length, $P$ the bytes per element, and the leading 2 counts keys and values. For a 70B-class model at 128k context this reaches tens of gigabytes. The crucial property is not its size but its access pattern: generating one token requires reading essentially the *entire* cache, while performing very little arithmetic on each byte fetched.

Arithmetic intensity explains everything. The roofline model says achievable performance is bounded by $\min(\text{peak FLOPs},\ I \times BW)$, where $I$ is arithmetic intensity in FLOPs per byte. Training's dense matrix multiplications have high $I$ and keep tensor cores saturated. Attention over the KV cache has $I$ below 1 — it is a streaming read with almost no reuse. That places decode far to the left of the roofline knee, where performance is set purely by $BW$. The practical consequence is blunt: in the decode phase, doubling memory bandwidth nearly doubles tokens per second, while doubling peak FLOPs changes almost nothing.

What LPDDR6 changes. LPDDR6 raises per-pin signalling to roughly 24 Gb/s against LPDDR5X's 9.6 Gb/s, restructures the channel into paired 48-bit channels with independent subchannels, and lowers operating voltage so that energy per bit improves rather than merely holding flat. It will not match HBM's per-stack bandwidth — HBM4 reaches about 2 TB/s per stack because it runs a 2048-bit bus across a silicon interposer — but it does not need a interposer, a TSV stack, or the packaging cost that comes with them. That makes LPDDR6 the memory of choice wherever bandwidth per dollar and bandwidth per watt matter more than absolute peak: phones, laptops, automotive inference, and the growing class of inference accelerators that cannot justify HBM economics.

MetricLPDDR5XLPDDR6HBM4
Per-pin data rate~9.6 Gb/s~24 Gb/s~8 Gb/s
Interface width16-bit channels24-bit × 2 subchannels2048-bit
Bandwidth per device/stack~77 GB/s~150+ GB/s~2 TB/s
PackagingStandard PoP / on-boardStandard PoP / on-board2.5D interposer + TSV
Relative cost per GBLowLowVery high
Typical usePhones, laptopsPhones, edge AI, inferenceTraining, frontier inference
LPDDR6 & the KV Cache — Why Inference Is a Memory Problem Every generated token re-reads the whole cache, so tokens/sec tracks bandwidth, not FLOPs The KV cache grows with every token tok 1 tok 2 tok 3 · · · tok N next token must read ALL every new token re-reads the entire cache from DRAM KV bytes = 2 × layers × heads × head_dim × tokens × precision 70B model, 128k context → tens of GB of cache, re-read per token Compute sits idle waiting on DRAM — this is the memory wall Arithmetic intensity here is under 1 FLOP per byte: pure streaming, no reuse LPDDR6 widens the pipe 24 Gb/s per pin (LPDDR5X: 9.6) 48-bit channel pairs, new subchannels lower V, better pJ/bit than LPDDR5X more bytes/sec per watt at the edge Cheaper and cooler than HBM no interposer, no TSV stack phones, laptops, edge inference PIM — stop moving the data at all Conventional: data travels to compute DRAM compute ~100× the energy of the math itself PIM: compute moves into the DRAM DRAM bank + MAC unit multiply-accumulate at the sense amps Samsung HBM-PIM, SK hynix AiM — bandwidth never leaves the die Three ways to attack the same wall HBM: widen the bus max bandwidth, max cost LPDDR6: cheaper bytes/watt edge and cost-sensitive inference PIM: move less data best pJ/bit, needs new software

Processing-in-memory — stop moving the data. Fetching a byte from DRAM into a compute unit costs on the order of a hundred times more energy than the arithmetic performed on it. PIM inverts the arrangement by placing multiply-accumulate units next to the DRAM banks themselves, at the sense amplifiers, so the enormous internal bandwidth of the array is consumed in place instead of being squeezed through the external pins. Samsung's HBM-PIM and SK hynix's AiM both implement this idea commercially. For attention workloads the fit is unusually good, because the operation is a streaming dot product over data that is already sitting in the array.

The obstacle is software, not silicon. PIM changes the programming model: the kernel must be expressed as operations the in-memory units can execute, data layout must match bank organisation, and the compiler has to decide what runs in memory versus on the accelerator. Until that toolchain is routine, PIM remains a specialised path rather than a drop-in bandwidth upgrade.

Three strategies, one wall. Each approach trades differently:

StrategyMechanismStrengthCost
HBM4Widen the bus to 2048-bit over an interposerHighest absolute bandwidthInterposer, TSVs, price, heat
LPDDR6Faster pins, better pJ/bit, standard packagingBest bandwidth per dollar and per wattFar below HBM in absolute terms
PIMCompute inside the DRAMBest energy per bit — data never movesNeeds a new programming model
SoftwarePaged/quantised KV cache, GQA, MLAShrinks the cache itselfAccuracy and complexity tradeoffs

The software lever is real. Because the KV cache is the thing being streamed, shrinking it is mathematically equivalent to widening the bus. Grouped-query attention reduces the number of key/value heads, cutting cache size several-fold. Multi-head latent attention compresses the cache into a lower-rank form. Quantising the cache to 8-bit or 4-bit halves or quarters it directly. PagedAttention avoids the fragmentation that wastes cache capacity. In practice these techniques often deliver a larger end-to-end speedup than a memory generation does, and they compose with it — a smaller cache on a wider bus multiplies.

What this means for system design. If a workload is dominated by long-context decode, the right questions are how many bytes of KV cache each token touches, whether that cache fits in the fastest memory available, and whether the attention variant can be changed to make it smaller. Peak FLOPs is close to irrelevant in that regime. LPDDR6 moves the achievable floor up for cost-sensitive inference hardware, HBM4 moves the ceiling up for frontier systems, and PIM proposes that the most efficient byte is the one that is never transferred at all.

Take lpddr6 & kv cache further

Ask the copilot about this term, or have our engineers assess it against your process.

Ask CFSGPT about this → Request a technical assessment → Open the CFS app