CXL 3.0 Memory Pool

CXL 3.0 Memory Pools: Coherent PCIe 6.0 Fabric and Multi-Host Sharing

Compute Express Link $(\mathrm{CXL}) 3.0$ and 3.1 are the architectural foundation for disaggregated, composable $\text{data-center}$ infrastructure. In AI and high-performance-computing clusters, static server memory allocation causes memory stranding: DRAM provisioned for one host's peak load sits idle most of the time, trapped behind that host's local CPU socket and unreachable by an adjacent $\text{memory-starved}$ accelerator — industry estimates commonly cite roughly a quarter to a third of provisioned cluster DRAM going unused this way, though the exact figure depends heavily on workload mix and provisioning policy.

CXL 3.0 addresses this by building an open, $\text{hardware-coherent}$ interconnect on top of the physical PCIe 6.0 PHY (64 GT/s PAM4). Symmetric $\text{multi-tiered}$ switching, $\text{fabric-attached}$ memory $(\mathrm{FAM})$, and $\text{hardware-enforced} \text{back-invalidation}$ cache coherency let multiple hosts dynamically share pooled memory without the host operating systems needing to coordinate the sharing in software. This is coherent $\cdot fabric^{\ast}$ access, not local DRAM access — it still crosses a switched PCIe link, so it adds real latency (commonly cited in the range of roughly 100-200+ ns over local DRAM, depending on topology and switch hop count) in exchange for pooling and sharing that a local memory bus cannot offer.

Physical & Protocol Architecture: Multi-Host Coherent Switching

The Fundamental Physics: PCIe 6.0 PAM4 Signaling & Cache Coherency

1. PCIe 6.0 PAM4 Physical Layer Dynamics

CXL 3.0 doubles raw signaling bandwidth from 32 GT/s (CXL 2.0 / PCIe 5.0) to 64 GT/s by migrating from Non-Return-to-Zero $(\mathrm{NRZ})$ to 4-Level Pulse Amplitude Modulation (PAM4).

In NRZ, each clock unit interval $(\mathrm{UI})$ transmits 1 bit using two voltage levels (0 and 1). In PAM4, each UI transmits 2 bits using four voltage levels (00, 01, 10, 11). However, subdividing the identical voltage swing ($V_{\text{peak}}-\text{to-peak} \approx 800$ mV) into three vertical eye openings reduces eye height by a factor of 3:

$V_{\text{eye}}$, PAM4 $= \dfrac{1}{3} V_{\text{eye}}, \mathrm{NRZ} \approx -9.54$ dB SNR penalty

To compensate for the reduced SNR and prevent bit errors from corrupting coherent memory lines:
* CXL 3.0 operates in a fixed $256-byte$ flit mode (flow control units) rather than the $\text{variable-length}$ transfers ordinary PCIe allows.
$\cdot A \text{low-latency}$ Forward Error Correction (FEC) engine works alongside cyclic redundancy check $(\mathrm{CRC})$, adding a small but real amount of latency (low tens of nanoseconds is the commonly cited order of magnitude for PCIe 6.0-class FEC, not a $\text{sub-nanosecond}$ figure) in exchange for a much lower residual error rate — published targets for the $\text{post-FEC}$ bit error rate are commonly quoted around:

$\mathrm{BER}_{\text{post}}-\mathrm{FEC} \le sssim 10^{-15}$ (target, not a guaranteed figure)

Memory Stranding: Static DIMMs vs. CXL 3.0 Pooling (Illustrative)

Protocol Comparison: CXL Generations

Why This Is Hard to Build

CXL 3.0 $\text{memory-expander}$ and PCIe 6.0 PAM4 retimer silicon is genuinely difficult: it combines $\text{fine-grained} \text{physical-layer}$ equalization (CTLE, DFE) at 64 GT/s with $\text{multi-host}$ coherency state machines that must stay correct under concurrent access from many hosts, and it must hold up across the cable and connector variety a real data center uses. That difficulty is a real barrier to entry, which is why a relatively small number of interconnect vendors are active in this space — but treat any specific list of winners, or any specific $\text{industry-wide}$ savings figure, as a claim to verify against current public data rather than something this article can state as settled fact.

CXL memory pooling belongs in the same family as the $\text{near-memory}$ and processing-in-memory architectures this series has covered: all of them are responses to data movement being more expensive than computing on data already in place. $\text{HBM-PIM}$, AiM, and UPMEM push compute down into memory to avoid moving data up to a processor; CXL pooling instead keeps memory and compute separate but makes the link between them coherent and shared, trading a local bus's latency for a fabric's flexibility. Which approach wins for a given system depends on whether the bottleneck is bandwidth to a fixed amount of nearby memory, or idle capacity stranded behind the wrong host.