inference chip

An inference chip is an accelerator optimized to run a trained neural network with low latency, high throughput, or low energy per request rather than to compute training gradients. **Serving is usually a data-movement problem.** Large-model decode repeatedly reads weights and a growing key-value cache for each generated token. HBM bandwidth, on-chip SRAM, batching strategy, and cache management can matter more than peak arithmetic throughput. **Designs specialize by deployment.** Data-center inference ASICs target efficient model serving at scale; edge chips emphasize INT8 and INT4 execution within tight power envelopes; GPUs retain flexibility across changing models and operators. | Constraint | Data center | Edge device | |---|---|---| | Primary goal | Tokens per second and latency | Energy and responsiveness | | Memory | HBM or large external DRAM | Shared mobile memory and SRAM | | Common precision | BF16, FP8, INT8, INT4 | INT8 and INT4 | | Typical workload | Large language and multimodal models | Vision, audio, and compact language models | **Software determines realized efficiency.** Quantization, graph fusion, continuous batching, speculative decoding, and a mature compiler/runtime stack often separate a useful inference product from an impressive peak specification.