What is NPU (Specialized processor for parallel tensor computations)

NPU (Neural Processing Unit) is a specialized hardware accelerator for neural networks. Unlike a universal central processor, NPU is physically designed for lightning-fast multiplication of multidimensional data arrays with minimal power consumption, imitating synaptic signal transmission directly during inference.

Such blocks are widely used in smartphones, wearable electronics, and smart cameras for real-time video processing, as well as in data centers for logical inference of large language models. NPU ensures instant face recognition, noise cancellation during calls, computational photography, and generative fill without the need for a constant connection to cloud servers.

The main challenge of implementation lies in the fragmentation of ecosystems and the lack of a universal programming standard: code optimized for Qualcomm will not run efficiently on Apple or Huawei chips without adaptation. In addition, the fixed architecture of quantized computations can lead to imperceptible degradation of model accuracy compared to graphics processors, as well as debugging difficulties due to the black box of proprietary drivers.

How NPU works

The operating principle is based on the architecture of systolic arrays and specialized on-chip memory, unlike the scalar cores of a CPU and the massively parallel streaming multiprocessors of a GPU. The central processor is designed for low latency of sequential instructions, while the graphics processor is designed for managing multiple pixel and vertex shaders with high-precision floating point. NPU sacrifices versatility for energy-efficient data flow through three-dimensional tensor pipelines without fetching complex instructions. Inside the chip, a network of multiply-accumulate cells is physically connected in cascades, where the result of one elementary computation is immediately transmitted to its neighbor, eliminating frequent accesses to RAM. The stream of weight coefficients synchronously pulses through the matrix of processor elements, turning the convolution or fully connected layer operation into a continuous hardware pipeline.

If a GPU simulates parallelism by launching thousands of threads on dozens of cores with context switching delays, then an NPU implements true spatially deterministic parallelism, where data is rigidly tied to the clock signal. Thanks to quantization to integer formats (INT8, INT4) and built-in activation blocks (ReLU, Sigmoid) directly in the datapath, inference speed increases manifold compared to vector extensions in a CPU, while heat dissipation remains within the limits of passive cooling for a mobile device.

NPU functionality

Low-precision matrix multiplication. NPU performs tensor operations on matrices of low bit width — INT8, INT4, and FP8 — in a single clock cycle. The systolic array of the processor breaks down convolution into many parallel MAC operations, minimizing register memory accesses and ensuring peak computational density per watt.
Hardware activation block. A specialized pipeline computes nonlinear functions — ReLU, Swish, GELU — without the intervention of scalar cores. The logic is embedded in the data path immediately after the matrix multiplier, eliminating extra transfers to shared memory and reducing layer output latency.
Weight loading pipelining. The direct memory access module uses a multidimensional DMA controller. It pulls weight tensors from external DRAM into the internal L1 buffer in parallel with current computations, overlapping bus latency and hiding the bottleneck of the von Neumann architecture.
DRAM (Storage and Byte-addressing of Data)
On-the-fly quantization. The preprocessing block converts incoming floating-point numbers into integer format immediately before entering the systolic array. Calibration scales are stored in dedicated accumulator registers, ensuring a dynamic range without losing throughput of the main pipeline.
Compression and sparse processing. The metadata decompressor skips zero weight values in real time, extracting only non-zero parameters from the packed bitstream. The structural sparsity engine powers down entire multiplier columns, radically reducing power consumption during inference of large language models.
Hardware graph branching. The command scheduler inside the NPU executes conditional transitions at the level of tensor operations without central processor involvement. This is critical for dynamic neural networks, where the graph shape changes depending on intermediate probability distributions or input sequence lengths.
In-memory pooling function. The post-processing block aggregates features by sliding a window over the feature map directly in the output buffer. The hardware max or average pooling adder extracts the dominant signal without write-read cycles, capturing translational invariance in negligible time.
Multi-cluster synchronization. A high-speed switching matrix connects several NPU clusters on the chip. The shared synchronization block implements a barrier model with hardware gradient reduction, allowing efficient parallelization of one giant model across several physical cores without cache coherence.
Zero-cost data transposition. The index permutation block is built into the crossbar interconnect of the buffers. The tensor does not physically move in memory but is logically reformatted from NHWC to NCHW format or vice versa when read by pointers, saving memory subsystem bandwidth resources.
Non-volatile sleep controller. The NPU power subsystem supports deep retention states with fast wake-up. Trigger blocks are activated within microseconds upon receiving an interrupt from audio sensors, preserving context in registers with ultra-low leakage, which is important for always-on scenarios.
Batch normalization as part of the weight. The hardware logic mathematically folds the parameters of the convolution kernel with the bias and scale parameters of the BN layer. Inference computation is performed using corrected coefficients in a single pass, completely eliminating a separate normalization stage from the execution pipeline.
Multiply-accumulate with increased precision. The internal accumulators of the systolic array operate with INT32 or FP32 format numbers. Despite the low bit width of input operands, the summation of partial products occurs without saturation and sign loss, guaranteeing numerical convergence of deep networks.
Feature map reuse. The local SRAM buffer is organized as a multi-bank register memory. Activation data consumed by several filters simultaneously is broadcast via a one-to-many port, radically reducing the number of accesses to energy-intensive external memory.
SRAM (Fast volatile random storage of bits)
Vector coprocessor core. In addition to the matrix engine, the NPU contains an SIMD operation block. It performs element-wise vector addition and multiplication operations, as well as nonlinearities that do not fit into dense multiplication, providing flexibility when performing Attention and Layer Normalization operations.
Vector (Ordered storage of numbers in continuous memory)
Streaming ECC controller. An error-correcting coding scheme is built into the DRAM interface. The mechanism corrects single-bit and detects double-bit errors on the fly without stopping the pipeline, which is critically important for the functional safety of neural computations in automotive ADAS systems.
ECC (Memory Error Detection and Correction)
Tensor hash accelerator. The module embeds features into sparse hash tables of large dimensions in hardware. Index computation and trainable embedding retrieval replace the fully connected layer, reducing the number of parameters in recommendation models by orders of magnitude.
Tensor (Multidimensional container for numerical data)
Frequency scaling without stream interruption. The clocking module dynamically changes voltage and frequency between network layer dispatches. Complex logic evaluates the thermal budget and utilization, adapting performance to a specific computational pattern without resetting the pipeline and preserving data integrity.
Hardware content protection. A zero-copy cryptographic engine decrypts the model weights from protected external memory into an internal isolated buffer. The host processor has no access to the keys and network parameters, which thwarts intellectual property cloning attacks.
Micro-batch buffer management. The task dispatcher splits the input tensor into tiles that fit into the dedicated SRAM. Hardware partitioning minimizes cache flushes, ensuring that even extremely large image resolutions are processed without throughput degradation due to buffer misses.

Comparisons

NPU vs CPU. The central processor is designed for the sequential execution of a wide range of tasks with high clock frequency and complex instruction flow control. The neural processor, on the contrary, has a massively parallel architecture with thousands of simple arithmetic cores optimized for low-bit-width matrix multiplication. This allows the NPU to process dense tensor computations an order of magnitude faster, consuming significantly less energy than the universal computing blocks of a CPU.
NPU vs GPU. The graphics processor, with its thousands of cores, was originally created for parallel processing of pixel shaders and textures; however, its programmable architecture has been successfully adapted for floating-point neural network training. The NPU differs by the hardware implementation of convolution operations and activation functions directly in silicon, eliminating extra clock cycles for instruction fetching, which provides lower latency and radically high energy efficiency at the inference stage.
NPU vs DSP. A digital signal processor is optimized for continuous streaming processing of analog-to-digital data with deterministic latency and a narrow specialization in multiply-accumulate operations. Unlike a DSP, a neural processor does not just perform MAC operations but contains a complex hierarchy of local memory and an on-chip network to minimize data movement when working with multidimensional tensors, which is critical for deep convolutional networks.
NPU vs FPGA. A field-programmable gate array allows implementing a perfect hardware architecture for a specific neural network topology, achieving minimal latency on non-standard operations and sparse computations. However, NPU wins due to the fixed, extremely dense layout of multiplication and accumulation blocks on the chip, which provides significantly higher clock frequency, lower power consumption, and ease of programming through high-level frameworks without circuit synthesis.
NPU vs TPU. The tensor processor from Google is a highly specialized ASIC design, tailored for the dominance of convolution and fully connected layer operations in server infrastructure using systolic arrays. The NPU in user devices is a more balanced system on a chip, supporting a wide range of operators, including recurrent networks and transformers, with a priority not on absolute peak performance, but on the maximum number of operations per watt within the thermal envelope of a smartphone.
TPU (Specialized accelerator for tensor multiplication operations)

OS and driver support

The interaction of the operating system with the NPU is realized through a specialized driver stack corresponding to frameworks such as OpenVINO, Qualcomm QNN, or Apple Core ML, where the kernel-mode driver manages memory allocation and task scheduling, and the user-mode driver translates high-level computation graphs into binary code understandable by the accelerator.

Security

Computation security is ensured by hardware isolation of NPU blocks via IOMMU (Input-Output Memory Management Unit), which prevents unauthorized access to system memory, and data models are protected by end-to-end encryption of weights and activations directly on the chip using a dedicated cryptographic coprocessor, excluding information interception on inter-chiplet buses.

IOMMU (Isolation of direct memory access addresses)

Logging and profiling

NPU operation monitoring is carried out through built-in hardware performance counters and specialized tracing APIs, which capture tensor core utilization, local SRAM bandwidth, and thermal throttling without stopping inference, transmitting structured metadata to the system log for bottleneck analysis by the task scheduler.

Hardware limitations

The key limitation of the NPU is the fixed architecture of the systolic array, optimized for low-bit-width integer operations, which makes the accelerator inefficient for tasks with irregular memory access, arbitrary control flow operations, and double-precision floating-point computations, requiring the hybrid execution of such instructions on a central or graphics processor.

Development history

The evolution of the NPU began with the emergence of highly specialized IP blocks for convolutional networks in mobile SoCs in 2014 and has progressed to programmable chiplets supporting transformers and dynamic matrix sparsity, facilitated by the adoption of the ONNX standard and hardware support for FP8 data types, which unified the deployment of large language models on accelerators from various vendors.