TOPS (Measuring peak performance of AI accelerators)

TOPS (Tera Operations Per Second) is a metric showing how many trillions of integer operations per second a chip can perform. The higher the number, the faster the neural network processes data, but actual speed depends on more than just this figure.

TOPS is widely used in the marketing positioning of neural processors in smartphones, laptops, and automotive chips. At the server level, the metric helps compare accelerators for data centers in computer vision and large language model tasks. Engineers also use it for an initial assessment of throughput when deploying trained networks.

Intensive computations often hit a bottleneck in the memory subsystem rather than peak arithmetic, causing a chip with high TOPS to idle. Manufacturers sometimes specify values for sparse computations or low INT4 precision, creating the illusion of a twofold advantage over a competitor working with dense INT8 data. Without accounting for power consumption and thermal envelope, raw performance does not guarantee stable operation without throttling.

How TOPS works

The principle of operation is based on multiplying the clock frequency by the number of available arithmetic logic units and the number of operations per clock performed by a specialized tensor core. Unlike teraflops, which are oriented toward floating-point operations on scalar values, TOPS reflects the integer parallelism critical for matrix multiplication in quantized neural networks. If a graphics accelerator can demonstrate high FP32 values for scientific calculations, an AI processor with an INT8 architecture will deliver a significantly higher TOPS number on the same die, since integer multiplication requires fewer transistors and energy. Modern neural processors supplement the classic MAC pipeline with activation blocks and pooling engines, striving to minimize pipeline stalls during typical computation graphs, including convolutions and attention mechanisms. This distinguishes the metric from the peak performance of central processors, where arithmetic instructions are interleaved with complex branching logic and cannot be loaded as densely over long inference runs.

TOPS functionality

1. Compute Bit Width. The metric is traditionally tied to the INT8 format. Multiply-accumulate is counted as two operations. Switching to 4-bit data types formally doubles the declared TOPS without changing the physical architecture of the chip.
2. Matrix Sparsity. Architectural support for structured sparsity doubles the declared throughput. The hardware block skips multiplication by zero, counting it as an operation, which creates a twofold gap between peak and real TOPS on dense models.
3. Matrix (Storing data in tabular form)
4. Clock Frequency Relationship. The TOPS number depends linearly on core frequency and the number of ALUs. The manufacturer multiplies the number of MAC units by two (multiply plus add), then by frequency, obtaining a theoretical ceiling without accounting for pipeline stalls and cache misses.
5. Memory-Bound Limitations. In autoregressive text generation tasks, the TOPS metric is useless. Performance hits the memory bandwidth wall, and compute blocks idle, making the peak value a purely marketing figure.
6. Vector Extensions. Modern NPUs implement systolic arrays whose configuration determines TOPS. A 128×128 matrix engine yields a different number of operations per clock than a 256×256 one, all else being equal, complicating direct architectural comparisons.
7. Vector (Ordered storage of numbers in continuous memory)
8. Energy Efficiency. A derived parameter is TOPS per Watt. It shows the number of operations per watt of consumed power, which is critical for passive cooling of edge devices and determines the SoC thermal budget during prolonged inference.
9. Activation Function. Nonlinearity operations (ReLU, SiLU) and normalization (LayerNorm) are often not included in the peak TOPS calculation. Hardware blocks implementing transcendental functions are accounted for separately, creating an additional bottleneck during actual model execution.
10. Tensor Virtualization. Compilation frameworks break down the computation graph into atomic instructions. TOPS utilization efficiency rarely exceeds 50 to 60 percent due to overhead from tensor reformatting and weight loading from external DRAM.
11. Tensor (Multidimensional container for numerical data)DRAM (Storage and Byte-addressing of Data)
12. Cluster Scaling. In multi-chip systems, declared TOPS is summed linearly. However, interconnects (NVLink, inter-socket bus) introduce latency, causing the effective number of operations to drop proportionally to the reduction coefficient during tensor parallelism.
13. Weight Quantization. Aggressive INT4 quantization doubles declared TOPS but requires calibration. Without outlier correction, model accuracy drops, forcing a return to INT8 computations and sacrificing half of the chip’s marketing performance.
14. Speculative Decoding. In assistant models, some operations are spent on discarded generation branches. Peak TOPS remains high, but useful work decreases, creating a gap between arithmetic block occupancy and the useful output token.
15. Batch Processing. TOPS utilization grows with batch size. During batch processing of requests in a data center, pipeline occupancy is maximized. On single inference requests typical of IoT, effective performance drops several times over.
16. Double Buffering. Asynchronous data prefetch via DMA into scratchpad memory helps mask DRAM access latency. Hardware delivers peak TOPS only when computations and I/O transactions are fully overlapped.
17. Latency Assessment. For time-critical tasks (autopilot), the execution time of a single operation is more important than peak throughput. Low TOPS on a chip with fast memory response is preferable to high TOPS with a slow cache hierarchy.
18. Role of Winograd Transformations. Transforming convolutions via the Winograd algorithm changes the nature of operations from matrix multiplication to element-wise. The TOPS metric is distorted because MAC units are used differently, and the number of useful MACs in the original network topology decreases.
19. MLPerf Benchmark. The industry standard measurement is throughput in queries per second on a specific model (ResNet, BERT). The TOPS figure on a datasheet correlates with the benchmark only with an identical memory architecture and the same software stack.
20. Structural Redundancy. Some ALUs are reserved for ECC protection mechanisms and safe AI. With memory fault protection enabled, effective TOPS decreases by the bandwidth amount allocated for error correction codes.
21. ECC (Memory Error Detection and Correction)

Comparisons

• TOPS vs FLOPS. TOPS assesses low-precision integer operations typical for neural network inference (INT8/FP8), while FLOPS measures floating-point operations critical for scientific calculations and high-precision model training. Direct comparison is incorrect, as one TOPS is not equivalent to one FLOPS due to radically different internal computation structures and data bit widths.
• TOPS vs MACs. Multiply-accumulate is the basic operation underlying neural network computations. The TOPS metric is functionally identical to TMACs (trillions of MACs per second) in sparse computations, but the 2-operations-per-MAC multiplier allows marketing to double the figure. The engineering value lies in the detail that accounts for matrix density and the architectural efficiency of compute block utilization.
• TOPS vs DL Boost (INT8) Score. The DL Boost metric characterizes the throughput of specific x86 VNNI instructions for accelerating convolutional networks on central processors. Unlike the universal TOPS metric, which measures the theoretical peak of a neural network accelerator as a whole, the DL Boost Score reflects the performance gain only during the inference stage with hardware instructions enabled, without accounting for the chip’s memory bandwidth.
• x86 (Execution of instructions based on CISC architecture)
• TOPS vs TOPS per Watt. The absolute TOPS value demonstrates peak computational power without considering power consumption and thermal constraints. The TOPS per Watt ratio introduces a fundamental correction for energy efficiency, becoming a key criterion for embedded and edge systems where the thermal envelope is strictly limited, and a drop in performance per watt negates the gain in raw computational power.
• TOPS vs Memory Bandwidth. The practical usefulness of peak TOPS values is blocked by the limited bandwidth of the memory subsystem. Even with an excess of arithmetic blocks, the accelerator can idle if the speed of loading weights and activations cannot keep up with the pace of computation. A balanced architecture requires sufficient bus width so that terabytes of data per second can saturate the arithmetic pipelines without scheduling bubbles.

Hardware foundation

The TOPS metric reflects the theoretical maximum of integer operations per second performed by a specialized neural coprocessor (NPU) or GPU tensor cores, and its implementation begins with drivers. The Windows operating system detects the NPU via the ACPI standard and loads a driver operating under the WDDM (Windows Display Driver Model) with a dedicated DirectML stack for machine learning, while in Linux, management goes through the DRM (Direct Rendering Manager) subsystem and the open-source OpenVINO framework, which compiles and distributes the computation graph among CPU, GPU, and NPU via the L0 plugin. The driver stack brings the hardware accelerator into an active state only upon receiving tasks from the framework, minimizing idle power consumption.

Computation isolation

Security when using neural accelerators is achieved through hardware isolation of command queues and sandboxes at the user-mode driver level, where each application operates in its own virtual address space and has no direct access to the buffers of other processes. This is implemented via IOMMU (Input-Output Memory Management Unit), which translates device virtual addresses to physical ones and prevents unauthorized DMA channel takeover, as well as through a protection model against Spectre-like attacks by flushing the NPU scheduler microarchitectural state upon context switching. During local processing of biometric data or confidential documents, the model is loaded into an encrypted memory region, and the central processor does not see the raw user data passing through the accelerator pipeline.

Operation tracing

Logging of TOPS operations is carried out by the driver through the Windows Event Tracing subsystem (ETW) or the ftrace mechanism in Linux, where the NPU provider records the execution duration of each forward pass, the load level of compute arrays, and memory bus bandwidth without saving the tensors themselves. Events are grouped by session identifiers, allowing the developer to reconstruct the picture: which model layers caused a drop in real performance relative to nominal TOPS, and the system logging daemon aggregates these events and creates an entry in the reliability log when speed falls below a threshold. User data does not end up in the log, as collection is performed only on kernel execution time metadata.

Limits

The main limitation of the metric is the impossibility of achieving declared TOPS on typical workloads without accounting for the intensity of access to external memory, as the accelerator encounters a bottleneck effect: compute blocks idle while the memory controller loads weights from LPDDR5X at a speed insufficient to fully saturate the array of MAC multipliers. Compensation is implemented through microarchitectural support for sparse computations, where the hardware scheduler skips zero weights and automatically reduces power consumption, but the final utilization coefficient for retaining peak performance rarely exceeds 40 to 65 percent on unstructured models. Additionally, computation precision switched to block FP16 or INT4 introduces error, making deterministic repeatability of the result unattainable on each run due to the non-commutativity of floating-point operations in parallel reduction.

Evolution

Historically, the term transitioned from evaluating the theoretical limit of GPUs without accounting for tensor cores to measuring exclusively the AI throughput of neural blocks, starting with the Qualcomm Snapdragon 855 SoC, where a metric of 7 TOPS for INT8 was explicitly stated for the first time. Development proceeded along the path of integrating NPUs into monolithic Apple Silicon chips with unified memory, eliminating copying between CPU and GPU, which raised real-world output to 40 TOPS when executing Core ML models, and then to heterogeneous Intel Meteor Lake accelerators with a dedicated VPU chiplet capable of processing up to 11 TOPS when powered via a USB-C bus. Current development focuses on the introduction of the Microsoft Pluton standard for verifying NPU firmware integrity before driver loading and the emergence of a unified WDDM 3.2 API, allowing the operating system to schedule AI tasks as critical system threads not preempted by the graphics queue.