Custom AI Accelerator Chips

AI Chip Landscape

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Groq LPU — Deterministic Inference at SRAM Speed</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">no HBM, no caches, no speculation — pure SRAM + compiler-scheduled dataflow = predictable latency</text>

  <!-- === LEFT: TSP Architecture === -->
  <rect x="30" y="65" width="330" height="235" rx="6" fill="#080d14" stroke="#233043" stroke-width="1.2"/>
  <text x="195" y="83" fill="#e6edf3" font-size="11" text-anchor="middle" font-weight="600">TSP (Tensor Streaming Processor)</text>

  <!-- Large SRAM block -->
  <rect x="50" y="95" width="290" height="55" rx="4" fill="#14261f" stroke="#34d399" stroke-width="1.4"/>
  <text x="195" y="116" fill="#6ee7b7" font-size="11" text-anchor="middle" font-weight="600">230 MB SRAM (on-die)</text>
  <text x="195" y="132" fill="#8b98a5" font-size="8.5" text-anchor="middle">no HBM, no DRAM — entire model weights in SRAM</text>
  <text x="195" y="144" fill="#6b7684" font-size="8" text-anchor="middle">80 TB/s internal bandwidth (vs 3.35 TB/s HBM on H100)</text>

  <!-- Functional units -->
  <rect x="50" y="158" width="135" height="35" rx="3" fill="#1c1633" stroke="#a78bfa" stroke-width="1"/>
  <text x="117" y="174" fill="#c4b5fd" font-size="9" text-anchor="middle" font-weight="600">VXM (Vector Unit)</text>
  <text x="117" y="187" fill="#8b98a5" font-size="7.5" text-anchor="middle">SIMD + scatter/gather</text>

  <rect x="195" y="158" width="135" height="35" rx="3" fill="#2a1a0a" stroke="#f59e0b" stroke-width="1"/>
  <text x="262" y="174" fill="#fbbf24" font-size="9" text-anchor="middle" font-weight="600">MXM (Matrix Unit)</text>
  <text x="262" y="187" fill="#8b98a5" font-size="7.5" text-anchor="middle">320×320 systolic</text>

  <!-- Streaming interconnect -->
  <rect x="50" y="200" width="290" height="22" rx="3" fill="#0f1a2a" stroke="#60a5fa" stroke-width="0.8"/>
  <text x="195" y="214" fill="#93c5fd" font-size="8.5" text-anchor="middle">Streaming interconnect (compiler-scheduled, no arbitration)</text>

  <!-- Control -->
  <rect x="50" y="228" width="290" height="22" rx="3" fill="#14202c" stroke="#8b98a5" stroke-width="0.8"/>
  <text x="195" y="242" fill="#8b98a5" font-size="8.5" text-anchor="middle">Deterministic execution: every cycle mapped at compile time</text>

  <!-- Key difference -->
  <text x="195" y="270" fill="#34d399" font-size="9" text-anchor="middle" font-weight="600">No cache misses. No stalls. No variability.</text>
  <text x="195" y="285" fill="#6b7684" font-size="8.5" text-anchor="middle">compiler does the scheduling (like VLIW), not hardware</text>

  <!-- === RIGHT TOP: Why it's fast for inference === -->
  <rect x="375" y="65" width="355" height="120" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="552" y="83" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Why Groq is Fast (Inference Only)</text>

  <text x="395" y="103" fill="#34d399" font-size="9">SRAM bandwidth: 80 TB/s (vs 3.35 TB/s HBM)</text>
  <text x="395" y="119" fill="#8b98a5" font-size="8.5">→ decode phase is memory-BW limited → 20x more BW = faster</text>
  <text x="395" y="139" fill="#34d399" font-size="9">Deterministic latency: every token same speed</text>
  <text x="395" y="155" fill="#8b98a5" font-size="8.5">→ no P99 tail latency spikes (great for real-time)</text>
  <text x="395" y="175" fill="#fbbf24" font-size="9">Result: ~500 tokens/s per user (Llama-3 70B)</text>
  <text x="395" y="188" fill="#6b7684" font-size="8">(vs ~80 tokens/s on H100 vLLM for same model)</text>

  <!-- === RIGHT BOTTOM: Limitations === -->
  <rect x="375" y="193" width="355" height="108" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="552" y="211" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Trade-offs</text>

  <text x="395" y="231" fill="#f87171" font-size="9">✗ Small memory (230 MB) — model must be distributed</text>
  <text x="395" y="247" fill="#8b98a5" font-size="8.5">  Llama-3 70B needs ~576 chips (memory-limited)</text>
  <text x="395" y="265" fill="#f87171" font-size="9">✗ No training — inference only (fixed dataflow)</text>
  <text x="395" y="281" fill="#f87171" font-size="9">✗ Limited model architectures (MoE harder)</text>
  <text x="395" y="297" fill="#6b7684" font-size="8.5">  Sweet spot: dense decoder-only LLMs at inference</text>

  <!-- === BOTTOM: Groq vs GPU vs TPU === -->
  <rect x="30" y="310" width="700" height="105" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="328" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">AI Accelerator Landscape (Inference)</text>

  <text x="130" y="350" fill="#76b900" font-size="9.5" text-anchor="middle" font-weight="600">NVIDIA H100</text>
  <text x="130" y="364" fill="#8b98a5" font-size="8" text-anchor="middle">80 GB HBM3</text>
  <text x="130" y="376" fill="#8b98a5" font-size="8" text-anchor="middle">general purpose</text>
  <text x="130" y="388" fill="#6b7684" font-size="7.5" text-anchor="middle">train + infer</text>

  <text x="290" y="350" fill="#34d399" font-size="9.5" text-anchor="middle" font-weight="600">Groq LPU</text>
  <text x="290" y="364" fill="#8b98a5" font-size="8" text-anchor="middle">230 MB SRAM</text>
  <text x="290" y="376" fill="#8b98a5" font-size="8" text-anchor="middle">fastest tokens/s</text>
  <text x="290" y="388" fill="#6b7684" font-size="7.5" text-anchor="middle">infer only</text>

  <text x="450" y="350" fill="#4285f4" font-size="9.5" text-anchor="middle" font-weight="600">Google TPU</text>
  <text x="450" y="364" fill="#8b98a5" font-size="8" text-anchor="middle">32-95 GB HBM</text>
  <text x="450" y="376" fill="#8b98a5" font-size="8" text-anchor="middle">systolic, JAX only</text>
  <text x="450" y="388" fill="#6b7684" font-size="7.5" text-anchor="middle">train + infer</text>

  <text x="610" y="350" fill="#f59e0b" font-size="9.5" text-anchor="middle" font-weight="600">Cerebras WSE</text>
  <text x="610" y="364" fill="#8b98a5" font-size="8" text-anchor="middle">40 GB SRAM (wafer)</text>
  <text x="610" y="376" fill="#8b98a5" font-size="8" text-anchor="middle">largest die ever</text>
  <text x="610" y="388" fill="#6b7684" font-size="7.5" text-anchor="middle">train + infer</text>

  <!-- Business -->
  <text x="380" y="410" fill="#8b98a5" font-size="9" text-anchor="middle">Groq: API available (GroqCloud), competitive pricing for inference-heavy workloads</text>

  <!-- Footer -->
  <text x="380" y="435" fill="#e6edf3" font-size="9.5" text-anchor="middle" font-weight="600">Groq's bet: for inference, memory bandwidth matters more than compute — so replace HBM with on-die SRAM</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Groq sacrifices generality for speed: a chip that does one thing (LLM decode) faster than anything else.</text>
</svg>

Groq LPU (Language Processing Unit)

Architecture

• Deterministic silicon: No caching, no variable latency
• SRAM-based: Large on-chip memory
• Tensor streaming: Optimized for sequential ops

Performance Claims

Groq API

from groq import Groq

client = Groq()

response = client.chat.completions.create(
    model="llama-3.2-90b-vision-preview",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)

Cerebras WSE (Wafer Scale Engine)

Unique Architecture

• Entire wafer as one chip (46,225 mm^2)
• 900,000 cores
• 40GB on-wafer memory
• Designed for massive models

Use Cases

• Training large models (no model parallelism needed)
• Drug discovery
• Climate modeling

Comparison

When to Consider

Best Practices

• Start with NVIDIA for flexibility
• Evaluate specialized hardware for specific needs
• Consider total cost (chips + development)
• Watch for SDK maturity
• Plan for vendor transitions