A "transformer chip" is silicon built to run transformer neural networks — the architecture behind GPT, Claude, and virtually every modern large language model — as fast and efficiently per token as possible. It is a family of accelerators, from data-center GPUs and TPUs to phone NPUs, organized around one insight: a transformer is mostly one operation done at enormous scale.

The workload is matrix multiplication. Attention computes $\text{Attention}(Q,K,V)=\text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$, while feed-forward layers are large linear projections. Most arithmetic is therefore dense matrix multiplication, so accelerators center on matrix engines such as NVIDIA Tensor Cores and Google systolic arrays rather than the scalar units that dominate a CPU.

Training and inference stress hardware differently. Training uses large batches and is usually compute-bound, rewarding raw throughput, fast interconnects, and lower precision. Autoregressive inference emits one token at a time while repeatedly reading weights and a growing key-value cache, so memory bandwidth often becomes the limit. A chip that excels at training is not automatically the most efficient serving chip.

The memory wall is the real fight. Decode performance depends on moving weights and KV-cache into matrix engines quickly. The leading answers are stacked HBM beside the compute die, advanced packaging such as TSMC CoWoS, large on-chip SRAM, and software such as FlashAttention that minimizes off-chip traffic. Packaging and memory capacity can constrain a useful accelerator more tightly than transistor count.

{ "rows": [
  { "type": "nodes", "items": [
    { "title": "Tokenize", "sub": "text to token IDs", "tone": "neutral" },
    { "title": "Embed", "sub": "vectors plus position", "tone": "neutral" }
  ] },
  { "type": "arrow" },
  { "type": "group", "title": "Transformer block", "note": "repeated every layer", "cycle": true, "loop": "stacks tens to hundreds of layers", "items": [
    { "title": "Attention", "sub": "Q K V matmuls", "tone": "green" },
    { "title": "Add and norm", "sub": "residual path", "tone": "green" },
    { "title": "Feed forward", "sub": "two big linears", "tone": "green" },
    { "title": "Add and norm", "sub": "residual path", "tone": "orange" }
  ] },
  { "type": "arrow" },
  { "type": "nodes", "items": [
    { "title": "Output head", "sub": "logits to next token", "tone": "orange" }
  ] }
] }

Precision keeps shrinking to buy throughput. FP32 gave way to FP16 and BF16, then FP8, while quantized INT8, INT4, and newer low-precision formats reduce inference memory traffic. Lower precision increases matrix throughput and moves fewer bytes, attacking both compute and bandwidth limits at once.

Transformer accelerators sit at the center of AI infrastructure. The workload is stable enough to reward specialized silicon, while access to accelerators — and to the HBM and advanced-packaging capacity behind them — has become strategically important.

A purpose-built transformer ASIC pushes this specialization further than a GPU can — the whole dataflow is fixed in silicon:

{ "rows": [
  { "type": "die", "title": "Transformer ASIC die", "note": "Dataflow fixed in silicon",
    "input": "Tokens",
    "top": [
      { "title": "Matmul array", "sub": "QKV and FFN projections", "tone": "green" },
      { "title": "Attention engine", "sub": "Fused softmax pipeline", "tone": "orange" }
    ],
    "base": { "title": "On-chip SRAM", "sub": "KV cache and activations", "tone": "blue" },
    "feed": { "title": "Off-chip HBM (weights)" }
  }
] }

Fixed dataflow, no scheduler. A GPU spends a large fraction of die area and power on being programmable — instruction decode, warp schedulers, register files, branch handling. A transformer ASIC hardwires the sequence (embed, QKV, attention, feed-forward, repeat), so nearly all transistors go to arithmetic. Etched claims its Sohu chip reaches more than 90 percent FLOPS utilization this way, versus the roughly 30 to 40 percent typical on GPUs, precisely because there is nothing to schedule.

Attention becomes a first-class pipeline. Instead of expressing attention as a chain of generic matmuls plus a separate softmax kernel, the whole QKᵀ, scale, softmax, times-V sequence is fused into one hardware pipeline. Intermediate scores never round-trip to memory — this is FlashAttention's insight, implemented in wires rather than CUDA.

The memory hierarchy is built for autoregressive decode. Inference is memory-bound: every generated token re-reads the KV cache and streams weights, so these chips go heavy on SRAM. Groq's LPU takes it to the extreme — no HBM at all, 230 MB of SRAM per chip, with models sharded across hundreds of chips in a deterministic, compiler-scheduled pipeline. That is how it reaches hundreds of tokens per second on 70-billion-parameter models. Cerebras does the wafer-scale version of the same idea, with 44 GB of SRAM on a single wafer.

Determinism falls out of the fixed dataflow. Because the dataflow is hardwired, execution time is known at compile time down to the cycle — no dynamic caches, no contention. That makes multi-chip pipelines trivially schedulable: the compiler is the network protocol.

The whole design space is a flexibility-for-efficiency trade. It runs roughly from the GPU (fully general), to the TPU (a systolic array, transformer-optimized but still programmable), to Groq and Cerebras (dataflow architectures), to Etched's Sohu (which can literally only run transformers). Each step trades flexibility for performance per watt. The obvious risk is architectural: if the field moves past transformers — state-space models like Mamba, hybrid attention schemes, whatever comes next — the most specialized chips become paperweights, which is why the hyperscalers hedge with TPU- and Trainium-style designs that keep a general matmul core.

Groq's LPU is the most interesting one to dig into — its "compiler owns everything, hardware does nothing clever" philosophy is almost the inverse of a GPU, and the ISCA papers on it are unusually readable. Happy to go deeper on the GPU-versus-LPU execution model, or on how the KV cache comes to dominate the memory budget during decode.