A neural processing unit (NPU) is a specialized processor designed to execute neural-network workloads with high energy efficiency. NPUs are commonly integrated into phones, PCs, vehicles, cameras, industrial systems, and other system-on-chip platforms so inference can run locally with lower latency, lower power, and less dependence on cloud connectivity.
Most neural-network execution is dominated by tensor operations such as matrix multiplication, convolution, attention projections, activation functions, normalization, and data rearrangement. An NPU accelerates the regular arithmetic with arrays of multiply-accumulate units, while local memories, interconnects, direct-memory-access engines, and control processors keep data moving through the array.
For a matrix product, the fundamental computation is:
The arithmetic itself is simple; the architectural challenge is reusing activations and weights enough times that data movement does not dominate energy and latency. Fetching a value from distant memory can cost far more energy than performing a low-precision multiply-accumulate, which makes the memory hierarchy and dataflow as important as the number of arithmetic units.
| NPU subsystem | Primary role | Design pressure |
|---|---|---|
| Tensor or MAC array | Executes dense and sparse dot products | Utilization across different tensor shapes |
| On-chip SRAM | Reuses weights, activations, and partial sums | Capacity, banking, bandwidth, and leakage |
| Network-on-chip | Moves tensors among engines and memories | Congestion, multicast efficiency, and power |
| Vector or scalar engine | Handles nonlinear and irregular operators | Operator coverage without excessive area |
| DMA and memory interface | Transfers data to and from system memory | Bandwidth, burst efficiency, and contention |
| Firmware and runtime | Schedules graphs and manages resources | Determinism, compatibility, and debug visibility |
Dataflow determines what stays close to the compute. Weight-stationary designs retain weights while activations move; output-stationary designs retain partial sums; row-stationary and hybrid approaches seek reuse across several dimensions. No single dataflow is optimal for every convolution, transformer projection, batch size, or sequence length, so modern NPUs often support tiling and scheduling flexibility.
Low-precision arithmetic is a major efficiency lever. INT8 is common for production inference, while INT4 and mixed-precision formats can reduce memory traffic and increase throughput when model accuracy permits. Some NPUs also support FP16, BF16, or floating-point variants for workloads that are difficult to quantize. Quantization-aware training or post-training calibration determines scales, zero points, clipping, and per-tensor or per-channel treatment. Advertised precision support is useful only when the compiler and kernels can execute the full graph efficiently at that precision.
Sparsity can avoid work and storage for zero-valued weights or activations, but its benefit depends on structure and hardware support. Unstructured sparsity may require metadata and irregular access that erase theoretical savings. Structured patterns are easier to schedule but constrain model training. A credible sparsity claim therefore states the pattern, encoding overhead, supported operators, and whether the published throughput assumes a particular zero rate.
The compiler is part of the architecture. A deployment stack imports a model, lowers operators to an intermediate representation, applies fusion and layout transformations, chooses precision, tiles tensors, allocates memory, and schedules kernels. Unsupported operators may fall back to the CPU or GPU. Those transitions can add copies, synchronization, and latency, so end-to-end graph coverage often matters more than peak kernel speed.
| Metric | What it can indicate | What it can hide |
|---|---|---|
| Peak TOPS | Maximum operations per second under stated assumptions | Precision, sparsity assumptions, and utilization |
| Sustained throughput | Performance on a defined workload | Batch size and quality constraints |
| Latency | Response time for a model or stage | Warm-up, transfer, and preprocessing time |
| Energy per inference | Battery or thermal efficiency | Accuracy, workload, and memory-system conditions |
| Operator coverage | Fraction of graph supported natively | Cost of fallback paths and graph partitioning |
TOPS is not a universal performance score. Vendors may count a multiply and an addition as two operations, quote different precisions, assume sparsity, or report a peak frequency that cannot be sustained thermally. Two devices with similar TOPS can perform very differently because of SRAM capacity, external bandwidth, tensor shapes, compiler quality, model partitioning, and power limits. Meaningful comparison uses the same model, accuracy target, batch size, precision, software version, and measured power boundary.
Edge NPUs are usually inference accelerators rather than general training processors. They may run vision, speech, sensor fusion, recommendation, language-model components, or generative features, but model size is constrained by memory and sustained power. Larger models can be tiled or streamed, although repeated external-memory access can reduce the efficiency advantage. Techniques such as distillation, pruning, quantization, and architecture search help fit models to the available hardware.
Silicon implementation is a balance of performance, power, area, and programmability. Larger arrays raise peak throughput but may sit idle on small or irregular workloads. More SRAM improves reuse but consumes die area and leakage power. Flexible routing and instruction support improve operator coverage but add control and verification complexity. Clock gating, power gating, voltage domains, thermal sensors, and quality-of-service controls help the NPU coexist with CPUs, GPUs, image processors, and modems on the same chip.
Production qualification includes functional verification across precisions and corner cases, numerical comparison against reference frameworks, memory-protection checks, fault handling, thermal and power validation, compiler regression testing, and model-level performance characterization. For safety-sensitive applications, teams may also require deterministic execution, diagnostic coverage, redundancy, or bounded-error analysis.
<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
<rect width="760" height="470" fill="#07131b"/>
<rect x="24" y="24" width="712" height="422" rx="18" fill="#10222e" stroke="#587989" stroke-width="1.2"/>
<text x="380" y="58" fill="#f5f8fa" font-size="20" font-weight="700" text-anchor="middle">NPU Dataflow</text>
<text x="380" y="82" fill="#93abb8" font-size="12" text-anchor="middle">keep tensors close to compute and minimize costly data movement</text>
<rect x="70" y="130" width="150" height="110" rx="10" fill="#17343f" stroke="#5bc8df"/>
<text x="145" y="173" fill="#b8edf7" font-size="13" font-weight="600" text-anchor="middle">System memory</text>
<text x="145" y="198" fill="#a5bdc7" font-size="10" text-anchor="middle">models and tensors</text>
<rect x="305" y="115" width="150" height="140" rx="10" fill="#17343f" stroke="#f2bd4d"/>
<text x="380" y="154" fill="#ffe09a" font-size="13" font-weight="600" text-anchor="middle">On-chip SRAM</text>
<text x="380" y="180" fill="#a5bdc7" font-size="10" text-anchor="middle">weights</text>
<text x="380" y="198" fill="#a5bdc7" font-size="10" text-anchor="middle">activations</text>
<text x="380" y="216" fill="#a5bdc7" font-size="10" text-anchor="middle">partial sums</text>
<rect x="540" y="130" width="150" height="110" rx="10" fill="#17343f" stroke="#6ed89d"/>
<text x="615" y="173" fill="#baf1cf" font-size="13" font-weight="600" text-anchor="middle">Tensor array</text>
<g fill="#2b674c" stroke="#8ce5b2"><rect x="570" y="192" width="20" height="20"/><rect x="596" y="192" width="20" height="20"/><rect x="622" y="192" width="20" height="20"/><rect x="648" y="192" width="20" height="20"/></g>
<path d="M220 185H305M455 185H540" stroke="#82aab8" stroke-width="3" marker-end="url(#arrow)"/>
<path d="M540 215H455" stroke="#82aab8" stroke-width="2" marker-end="url(#arrow)"/>
<rect x="90" y="305" width="580" height="78" rx="10" fill="#0e1f28" stroke="#567786"/>
<text x="380" y="333" fill="#a9c0c9" font-size="12" font-weight="600" text-anchor="middle">Compiler and runtime schedule the graph</text>
<text x="380" y="357" fill="#91aab5" font-size="10" text-anchor="middle">fusion, quantization, tiling, memory allocation, operator placement, and fallback</text>
<text x="380" y="414" fill="#91aab5" font-size="10" text-anchor="middle">sustained efficiency depends on utilization, bandwidth, precision, and graph coverage</text>
<defs><marker id="arrow" viewBox="0 0 10 10" refX="8" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path d="M0 0L10 5L0 10Z" fill="#82aab8"/></marker></defs>
</svg>
In short, an NPU is not merely a large array of multipliers. Its real efficiency comes from co-designing arithmetic, memory, dataflow, interconnect, compiler, runtime, and models so useful neural-network work remains on-chip and highly utilized under the product's power and latency limits.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.