chip architecture

A **transformer chip** is silicon built to run transformer neural networks — the architecture behind GPT, Claude, and virtually every modern large language model — as fast and efficiently per token as possible. It is a family of accelerators, from data-center GPUs and TPUs to phone NPUs, organized around one insight: a transformer is mostly one operation done at enormous scale. The diagram below is the anatomy every one of these chips is arguing about — where the arithmetic happens, and why the path from memory to that arithmetic is the real battleground.\n\n```svg\n\n```\n\n**The workload is matrix multiplication.** Attention computes $\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$, while feed-forward layers are large linear projections. Most arithmetic is therefore dense matrix multiplication, so accelerators center on matrix engines such as NVIDIA Tensor Cores and Google systolic arrays rather than the scalar units that dominate a CPU.\n\n**Training and inference stress hardware differently.** Training uses large batches and is usually compute-bound, rewarding raw throughput, fast interconnects, and lower precision. Autoregressive inference emits one token at a time while repeatedly reading weights and a growing key-value cache, so memory bandwidth often becomes the limit. A chip that excels at training is not automatically the most efficient serving chip.\n\n**The memory wall is the real fight.** Decode performance depends on moving weights and KV-cache into matrix engines quickly. The leading answers are stacked HBM beside the compute die, advanced packaging such as TSMC CoWoS, large on-chip SRAM, and software such as FlashAttention that minimizes off-chip traffic. Packaging and memory capacity can constrain a useful accelerator more tightly than transistor count.\n\n| Chip family | Example | Best at | Core engine |\n|---|---|---|---|\n| Data-center GPU | NVIDIA H100 and Blackwell | Flexible training and serving | Tensor Cores plus HBM |\n| TPU | Google TPU | Dense matrix math at scale | Systolic array |\n| Inference ASIC | AWS Inferentia and Groq LPU | Efficient serving | Specialized dataflow |\n| Edge NPU | Phone and laptop NPU | On-device inference | INT8 and INT4 MAC array |\n| Transformer ASIC | Emerging dedicated designs | Narrow transformer workloads | Hardwired tensor dataflow |\n\nThe logical dataflow the silicon has to serve — tokens in, a stack of identical blocks, logits out:\n\n```flowchart\n{ "rows": [\n { "type": "nodes", "items": [\n { "title": "Tokenize", "sub": "text to token IDs", "tone": "neutral" },\n { "title": "Embed", "sub": "vectors plus position", "tone": "neutral" }\n ] },\n { "type": "arrow" },\n { "type": "group", "title": "Transformer block", "note": "repeated every layer", "cycle": true, "loop": "stacks tens to hundreds of layers", "items": [\n { "title": "Attention", "sub": "Q K V matmuls", "tone": "green" },\n { "title": "Add and norm", "sub": "residual path", "tone": "green" },\n { "title": "Feed forward", "sub": "two big linears", "tone": "green" },\n { "title": "Add and norm", "sub": "residual path", "tone": "orange" }\n ] },\n { "type": "arrow" },\n { "type": "nodes", "items": [\n { "title": "Output head", "sub": "logits to next token", "tone": "orange" }\n ] }\n] }\n```\n\n**Precision keeps shrinking to buy throughput.** FP32 gave way to FP16 and BF16, then FP8, while quantized INT8, INT4, and newer low-precision formats reduce inference memory traffic. Lower precision increases matrix throughput and moves fewer bytes, attacking both compute and bandwidth limits at once.\n\n**A purpose-built transformer ASIC pushes specialization further than a GPU can.** The whole dataflow is fixed in silicon. A GPU spends a large fraction of die area and power on being programmable — instruction decode, warp schedulers, register files, branch handling. A transformer ASIC hardwires the sequence (embed, QKV, attention, feed-forward, repeat), so nearly all transistors go to arithmetic. Etched claims its Sohu chip reaches more than 90 percent FLOPS utilization this way, versus the roughly 30 to 40 percent typical on GPUs, precisely because there is nothing to schedule.\n\n**Attention becomes a first-class pipeline.** Instead of expressing attention as a chain of generic matmuls plus a separate softmax kernel, the whole $QK^\top$, scale, softmax, times-$V$ sequence is fused into one hardware pipeline. Intermediate scores never round-trip to memory — this is FlashAttention's insight, implemented in wires rather than CUDA.\n\n**The memory hierarchy is built for autoregressive decode.** Inference is memory-bound: every generated token re-reads the KV cache and streams weights, so these chips go heavy on SRAM. Groq's LPU takes it to the extreme — no HBM at all, 230 MB of SRAM per chip, with models sharded across hundreds of chips in a deterministic, compiler-scheduled pipeline. That is how it reaches hundreds of tokens per second on 70-billion-parameter models. Cerebras does the wafer-scale version of the same idea, with 44 GB of SRAM on a single wafer.\n\n**Determinism falls out of the fixed dataflow.** Because the dataflow is hardwired, execution time is known at compile time down to the cycle — no dynamic caches, no contention. That makes multi-chip pipelines trivially schedulable: the compiler is the network protocol.\n\n**The whole design space is a flexibility-for-efficiency trade.** It runs roughly from the GPU (fully general), to the TPU (a systolic array, transformer-optimized but still programmable), to Groq and Cerebras (dataflow architectures), to Etched's Sohu (which can literally only run transformers). Each step trades flexibility for performance per watt. The obvious risk is architectural: if the field moves past transformers — state-space models like Mamba, hybrid attention schemes, whatever comes next — the most specialized chips become paperweights, which is why the hyperscalers hedge with TPU- and Trainium-style designs that keep a general matmul core.\n\nRead a transformer chip through a *bandwidth* lens rather than a *FLOPS* lens: the number that sets tokens-per-second-per-dollar is how fast weights and KV-cache reach the matrix engines, not the peak arithmetic rate printed on the datasheet. Every design in this space — HBM versus all-SRAM, GPU versus hardwired ASIC, FP16 versus INT4 — is ultimately a different answer to the same question of how to keep the matmul units fed.\n