Neural Processing Unit
A neural processing unit (NPU) is a specialized processor block, usually integrated into a phone, laptop or embedded system-on-a-chip, designed to run the multiply-accumulate-heavy workloads of neural networks at low power. Where a GPU is a large, flexible parallel processor and a data-center accelerator is built for maximum throughput, an NPU is built for efficiency within a tight power and thermal budget, so that tasks such as face recognition, speech transcription, photo enhancement, background blur in video calls, and increasingly small language models can run locally without draining the battery or sending data to a server.
The modern NPU era began in 2017, when phone makers started putting dedicated neural hardware beside the CPU and GPU. Apple's A11 Bionic, used in the iPhone 8 and iPhone X, introduced the Apple Neural Engine, a two-core block that Apple described as capable of 600 billion operations per second and which powered Face ID and Animoji. Huawei's Kirin 970, also announced in 2017, included an NPU developed with Cambricon, and Google's Pixel 2 shipped a separate Pixel Visual Core for image processing. Qualcomm's Hexagon DSP gained tensor acceleration, and later Qualcomm, Apple, Google (with Tensor chips in Pixel phones from 2021), Samsung, MediaTek and others made the NPU a standard part of mobile silicon. In PCs the idea arrived later: Intel's Meteor Lake processors (December 2023) and AMD's Ryzen AI parts included NPUs, and in May 2024 Microsoft defined its Copilot+ PC category with a requirement of an NPU rated at 40 TOPS or more.
Efficiency comes from architectural choices that trade generality for energy per operation. NPUs typically use large arrays of low-precision multiply-accumulate units, favoring INT8 and sometimes INT4 or FP16 arithmetic, with on-chip scratchpad memory arranged so that weights and activations move as little as possible, since moving data off-chip costs far more energy than the arithmetic itself. They are optimized for fixed dataflows such as convolutions and matrix multiplications and are managed by a compiler or runtime that maps a model onto the hardware. Most of the machinery is shared with data-center designs, such as systolic arrays and quantization, but with energy measured in milliwatts to a few watts rather than hundreds of watts.
The TOPS figure printed on spec sheets is the most misleading number in the category. TOPS means trillions of operations per second, and it is a theoretical peak that depends on conventions: it is usually quoted for low-precision integer arithmetic such as INT8, counts a multiply and an add as two operations, and may include structured sparsity that real models do not exploit. A chip's TOPS rating at INT8 cannot be compared with another vendor's at INT4, with a GPU's FP16 figure, or with sustained performance. Real throughput is often limited by memory bandwidth, particularly for large language models, whose token generation is memory-bound, and by whether the software stack supports a given operator. Two NPUs with equal TOPS can differ severalfold on an actual model, so benchmarks such as MLPerf Mobile and MLPerf Client, and measurements on the target workload, say more than the headline.
Software support is where on-device AI succeeds or stalls. Models usually need to be converted, quantized and sometimes restructured to run on an NPU, through toolchains such as Apple's Core ML, Google's LiteRT (formerly TensorFlow Lite), ONNX Runtime with vendor execution providers, and Qualcomm's AI Engine. Operators the NPU does not implement fall back to the CPU or GPU, and the transfers can erase the benefit. Running language models on-device generally means small or aggressively quantized models, because a phone's shared memory must hold the whole model alongside everything else. The payoff is real: lower latency, offline operation, no per-query server cost, and a privacy benefit because raw audio, photos or text need not leave the device, though privacy depends on the application and is not guaranteed by the hardware itself.
A frequent misconception is that an NPU makes a device able to run any AI model, or that it replaces the GPU. Large frontier models still require data-center ai-accelerator hardware with far more memory and bandwidth. NPUs complement the CPU and GPU, and each application decides which unit executes each part of its model. The sensible question for a buyer or developer is which concrete features use the NPU, at what precision, and with what measured latency and battery cost.
Read the neural processing unit as a hearing aid for software rather than a supercomputer: a small, frugal circuit tuned to do one narrow class of arithmetic constantly and cheaply, whose advertised TOPS tell much less than the models and software that actually run on it.