FlashAttention Kernel Simulator

Model why FlashAttention is the kernel behind long-context Transformers — then run it: the simulation executes on the ChipFoundryServices compute pool. Standard attention writes the full S×S scores matrix to HBM, so its memory traffic grows quadratically with sequence length — and softmax reads it back several times, leaving the tensor cores starved. FlashAttention tiles Q, K and V into blocks that fit in on-chip SRAM and runs an online softmax, so it never materializes that matrix: its HBM traffic is linear in S (just Q, K, V and O), and the reduction naive / FA grows as S / (2·head_dim). On the roofline, cutting the bytes lifts arithmetic intensity above the machine balance, flipping attention from memory-bandwidth-bound to compute-bound at the same FLOPs — the whole point. The tiles must fit in shared memory (a block of B needs about 4·B·head_dim·bytes); too large a block, head dim or precision overflows SRAM and cuts occupancy. The node returns the HBM traffic for both kernels, the S×S matrix FlashAttention avoids, the arithmetic intensity, the bottleneck and the wall-clock speed-up. Reduced-order educational model. See also the quantization, KV-cache, HBM bandwidth, systolic array, 3D-parallelism, training-memory, power & thermal, transistor I-V, thermal, die-yield and lithography simulators and the compute-pool status.

Long context · 32k Short seq · 512 FP8 · H100 SRAM overflow
Roofline — performance ceiling vs arithmetic intensity (FLOPs per HBM byte), log-log. The diagonal is the memory-bandwidth limit, the flat line the compute ceiling, and the ridge point where they meet is machine balance. Naive attention sits far left (low intensity, memory-bound); FlashAttention keeps the same FLOPs but reads far fewer bytes, sliding right past the ridge into the compute-bound regime
Left: HBM traffic reduction vs sequence length — naive attention materializes the S×S score matrix (quadratic bytes), FlashAttention never writes it (linear bytes), so the reduction grows with context. Right: HBM traffic per kernel — FlashAttention vs naive as a share of the naive total; ◂ marks the score-matrix traffic FlashAttention avoids
Developer API — same simulation over HTTP (load-balanced across the pool): 
curl -X POST https://www.chipfoundryservices.com/edge/flashattention \
  -H "Content-Type: application/json" \
  -d '{"batch_size":8,"num_heads":32,"seq_length":32768,"head_dim":128,
       "elem_bytes":2,"causal":1,"sram_kb":192,"block_size":128,
       "hbm_bandwidth_gb_s":3350,"compute_tops":990}'
Returns JSON with outputs (format, achieved_block, max_block_sram, sram_needed_kb, sram_used_kb, sram_budget_kb, sram_fit, q_blocks, kv_blocks, flops_gflop, scores_matrix_gb, naive_hbm_gb, fa_hbm_gb, hbm_reduction_x, machine_balance_flop_byte, arithmetic_intensity_fa, arithmetic_intensity_naive, fa_bottleneck, naive_bottleneck, naive_time_ms, fa_time_ms, compute_time_ms, speedup_x, fa_tflops, fa_util_percent, verdict), the full profile (48-point reduction_vs_seq HBM-reduction and speedup_vs_seq wall-clock-speedup sweeps, plus ai_fa, ai_naive, machine_balance, peak_tflops, hbm_bw_gb_s, hbm_reduction_x, seq_length, speedup_x), the serving node, and compute_ms. Endpoint aliases /edge/flash, /edge/flashattn, /edge/fa, /edge/attn, /edge/sdpa.