KV-Cache / Attention-Memory Simulator

Model the attention KV cache of a Transformer being served — the Key/Value tensors an LLM caches for every past token so decode never recomputes them — then run it: the simulation executes on the ChipFoundryServices distributed compute pool. From the model geometry (layers × KV heads × head dim) it computes the per-token cache cost, then the total HBM footprint (weights + cache) for your context length and batch, whether it fits in HBM or OOMs, the capacity headroom (max batch, max context), and rolls it up to the headline number: the memory-bound decode throughput. Every generated token re-reads the whole KV cache, so at long context the cache — not the weights — gates throughput, which is why long-context serving is a memory problem, not a FLOP problem. Grouped-query attention (fewer KV heads) and KV quantization are the main levers. Reduced-order educational model. See also the HBM bandwidth, systolic array, transistor I-V, thermal, interconnect RC, die-yield, 6T SRAM, CMP planarization and lithography simulators and the compute-pool status.

Llama-3-8B · H100 Llama-3-70B · 8×H100 Long-context · 128K
HBM footprint vs context length (log–x) — weights (dashed baseline) are fixed; the KV cache grows linearly with context and batch until it hits the capacity ceiling and the request OOMs. Your context (●) sits on the curve; the gap up to the roof is your remaining headroom
Left: decode throughput vs context length — every token re-reads the whole KV cache, so throughput collapses as context grows. Right: the capacity budget — weights + KV cache as a share of HBM. Marker is your context
Developer API — same simulation over HTTP (load-balanced across the pool): 
curl -X POST https://www.chipfoundryservices.com/edge/kvcache \
  -H "Content-Type: application/json" \
  -d '{"model_params_b":8,"num_layers":32,"num_kv_heads":8,"head_dim":128,
       "context_length":8192,"batch":32,"kv_precision_bits":16,
       "weight_precision_bits":16,"hbm_capacity_gb":80,"hbm_bw_gbs":3350}'
Returns JSON with outputs (kv_bytes_per_token, kv_kb_per_token, kv_per_sequence_gb, kv_total_gb, weight_gb, footprint_gb, capacity_gb, utilization_percent, kv_fraction_percent, free_for_kv_gb, fits, max_batch_at_context, max_context_at_batch, bytes_per_step_gb, time_per_step_ms, decode_tokens_per_s, bottleneck, verdict), the full profile (48-point footprint_vs_ctx and tps_vs_ctx sweeps, plus weight_gb, kv_total_gb, capacity_gb, kv_bpt, kv_frac, bottleneck), the serving node, and compute_ms. Endpoint aliases /edge/kv, /edge/attention, /edge/serving, /edge/context.