NPU
How it works
An NPU maps neural-network layers onto MAC-optimized hardware. Its core is a systolic array or a grid of MAC units through which data (activations and weights) flow rhythmically, performing multiply-accumulate without constant instruction fetching. Weights and activations are quantized to low precision (usually INT8), multiplying the operations per cycle and reducing memory footprint. Local SRAM buffers weights and intermediate results, while a DMA/dataflow controller orchestrates data movement to minimize costly external-memory access. A compiler/runtime (e.g., supplied by the SoC vendor) translates a model from a framework (PyTorch/TensorFlow/ONNX) into an operation graph, quantizes it, and schedules it onto NPU resources. Performance is bounded by the number of MAC units (and frequency) and by memory bandwidth; real-world capability is quoted in TOPS and TOPS/W.
Problem solved
Neural networks require enormous numbers of matrix multiplications; running them on a CPU is slow, and on a GPU power-hungry and costly, especially on mobile and edge devices with tight power budgets. The NPU provides hardware dedicated to these operations, offering high throughput at low power and enabling AI inference locally, in real time.
Components
A grid of multiply-accumulate units performing matrix multiplications and convolutions; data flows through the array rhythmically (systolic dataflow).
Fast local memory buffering weights, activations, and intermediate results, cutting costly external-memory access.
Controls the flow of weights and activations between memory and the MAC array, scheduling operations without CPU involvement.
Hardware support for low-precision arithmetic (INT8/INT4) and for activation, normalization, and pooling functions.
Implementation
Quantizing to INT8/INT4 can degrade model quality without calibration or quantization-aware training (QAT).
Many models never reach peak TOPS because they are limited by weight/activation transfer, not compute.
NPUs support a limited operator set; unusual layers fall back to CPU/GPU, negating the performance gains.
Evolution
Google deploys a systolic-array accelerator (TPU) for datacenter inference, popularizing dedicated neural-network accelerators.
Huawei's Kirin 970 and Apple's A11 (Neural Engine) bring NPUs to mainstream mobile devices, launching the on-device AI era.
Energy-efficient edge NPUs (e.g., Google Edge TPU) emerge, enabling inference in IoT and robotics.
Intel, AMD, and Qualcomm add NPUs to PC processors; the Copilot+ PC platform sets a performance bar (tens of TOPS) for local AI.
Hyperparameters (configurable axes)
The number of MAC units and frequency set peak throughput (TOPS), the NPU's key performance parameter.
Supported low-precision formats; lower precision raises TOPS and cuts memory at the cost of accuracy.
Memory bandwidth and on-chip SRAM size are often the real bottleneck (roofline), not the raw MAC count.
Execution paradigm
Spatial/dataflow execution dominated by dense matrix multiplication; newer NPUs also support sparsity for extra efficiency.
Parallelism
An NPU achieves spatial parallelism in the MAC array: hundreds to thousands of multiplications per cycle under a dataflow execution model.
Hardware requirements
The TPU is the canonical datacenter NPU built on a systolic array โ a direct realization of this concept.
GPU tensor cores perform the same low-precision MAC operation; functionally similar to an NPU.
An NPU can be implemented on an FPGA, though with lower efficiency than a dedicated ASIC.