what is an npu
An NPU, or Neural Processing Unit, is a chip or chip block built specifically to run the repetitive math behind neural networks — mostly large batches of matrix multiplication — far more efficiently than a general-purpose CPU or even a GPU can, which is why NPUs have become a standard part of phones, laptops, and cars that need to run AI features directly on the device rather than sending every request to the cloud.
```flowchart
{
"rows": [
{ "type": "nodes", "items": [
{ "title": "Neural network inference", "sub": "mostly matrix multiplication, repeated millions of times", "tone": "neutral" }
]},
{ "type": "arrow" },
{ "type": "group", "title": "Three chips could run it", "items": [
{ "title": "CPU", "sub": "flexible, general-purpose, least efficient for this", "tone": "blue" },
{ "title": "GPU", "sub": "broadly parallel, better, still general-purpose", "tone": "orange" },
{ "title": "NPU", "sub": "built only for this math, most efficient", "tone": "green" }
]},
{ "type": "arrow" },
{ "type": "nodes", "items": [
{ "title": "On-device AI: fast, private, power-efficient", "sub": "no round trip to a data center required", "tone": "green" }
]}
]
}
```
**An NPU is efficient because it throws away everything a neural network's math doesn't need.** A CPU is built to run wildly varied, unpredictable instructions one after another, with lots of circuitry dedicated to handling branches and decisions; an NPU instead is built almost entirely around a narrow, well-understood pattern — take numbers in, multiply and add them in enormous batches, and pass the results forward — allowing far more of its transistor budget and power to go directly into that one repeated operation instead of general-purpose flexibility it will never use for this task.
```svg
```
**The main reason phones and laptops now ship with a dedicated NPU is power efficiency, not raw speed.** A GPU can often run a neural network just as fast as an NPU, but it typically draws far more power doing it; a battery-powered device running AI features continuously — voice processing, photo enhancement, live translation — needs that work done using as little energy as possible, and a chip narrowly designed around exactly that math delivers dramatically more inference per watt than a flexible, general-purpose alternative.
| Chip | Flexibility | Efficiency for neural network math | Typical role |
|---|---|---|---|
| CPU | Highest — runs anything | Lowest for this specific workload | General computing, light AI tasks |
| GPU | High — broad parallel workloads | Good — much better than CPU | Training and heavy inference, often in data centers |
| NPU | Lowest — built for one job | Highest per watt for neural network math | On-device inference: phones, laptops, cars |
**Running AI directly on the device instead of in the cloud also solves problems that raw speed alone can't fix — privacy and latency.** Sending a photo, a voice recording, or personal text to a remote server for every AI feature means that data has to leave the device and travel over a network before a response comes back; an NPU lets that same inference happen locally, in milliseconds, without the data ever leaving the device — which matters for private information and for features that need to feel instantaneous.
**NPUs are built almost entirely for inference, not training, which is an important distinction when comparing them to data-center AI chips.** Training a large neural network from scratch requires enormous amounts of compute, memory bandwidth, and iterative adjustment across huge datasets — work still done predominantly on large GPU or specialized AI-training chips in data centers. An NPU's job is running an already-trained model efficiently on new input, a much narrower and more predictable task that needs far less raw compute and memory.
Read the NPU through an efficiency-for-one-job lens: it isn't trying to be the fastest or most flexible chip in the device, it's trying to run one specific, extremely common kind of math — the arithmetic behind a trained neural network — using as little power as possible, which is exactly the tradeoff a battery-powered device running AI features all day actually needs.