TPU (Specialized accelerator for tensor multiplication operations)

TPU is a specialized Google processor created for ultra-fast execution of matrix computations. It is a hardware accelerator that sacrifices the versatility of a central processor for the phenomenal speed of multiplying arrays of numbers that form the basis of neural networks.

The main application area of TPUs is large-scale machine learning and inference in Google data centers. They underlie services such as search, translator, voice recognition, and routing in Google Maps. External users also rent TPUs through the Google Cloud platform for training transformers, large language models, and generative neural networks that require processing huge batches of data with low latency.

The main limitation is extremely narrow specialization and dependence on the TensorFlow or JAX ecosystem, which makes code written for PyTorch non-portable without adaptation. Programming under the tensor paradigm requires careful selection of matrix dimensions, otherwise a significant portion of computational power remains idle. There are also debugging difficulties due to the closed nature of the architecture, and memory bandwidth can become a bottleneck when working with unstructured sparse data.

How a TPU works

The TPU architecture is fundamentally different from CPUs and GPUs due to the use of a systolic array: a grid of thousands of arithmetic units that rhythmically pass data to each other without accessing registers or cache memory. Unlike a central processor optimized for complex instruction flow control and executing sequential operations at high clock speeds, the TPU is designed as a state machine driven by a deterministic sequence of CISC instructions from the host. The graphics processor, although possessing thousands of cores, spends significant energy on flexible thread scheduling and cache memory hierarchy, whereas the TPU implements matrix multiplication directly in the silicon of the systolic array. When performing a convolution or matrix multiplication operation, input parameters are loaded from fast memory with high bandwidth and pumped through the array from left to right and top to bottom, accumulating partial sums without storing intermediate results in shared memory. This allows achieving extreme energy efficiency and teraflops performance while avoiding energy-intensive read-write transactions characteristic of the von Neumann architecture.

TPU functionality

1. Systolic array architecture. The core of the computational unit is a systolic matrix MXU with dimensions of 128x128 or 256x256. Data pulses through an array of multiply-accumulate units rhythmically, without the need to access a register file or cache on every cycle, minimizing energy costs for instruction fetching.
2. INT8 and BF16 quantization. The hardware unit provides native execution of matrix multiplication with 8-bit integer precision, achieving peak performance. The chip natively supports the Brain Floating Point 16 format, preserving the dynamic range of FP32 through a truncated mantissa for training tasks with minimal accuracy degradation.
3. BF16 (Floating-point number format for machine learning)
4. Deterministic instruction flow. The TPU operates not as a coprocessor but as a dedicated accelerator controlled via the PCIe bus. The host processor sends a binary computation graph to the queue buffer. The device executes CISC-like instructions linearly, eliminating speculative execution, branch prediction, and out-of-order loading.
5. Dedicated activation buffer. A unified buffer of 24–48 MiB is placed on the chip, serving as RAM for storing intermediate tensors. Data locality in this buffer avoids accessing the energy-expensive off-chip HBM memory at every stage of the operation graph pipeline.
6. HBM (3D stacked memory with silicon vias)
7. Saturation arithmetic. In INT8 integer mode, saturation arithmetic is used rather than modular arithmetic. On overflow, the operation result is forcibly fixed at the boundary of the permissible range, which is fundamentally important for maintaining the stability of quantized neural network weights during inference without generating exceptions.
8. On-the-fly weight transposition. A specialized input data preprocessing module performs hardware matrix rearrangement to optimize read patterns. This frees the programmer from manual data layout and allows the systolic array to always fetch coefficients with maximum bandwidth without pipeline idle cycles.
9. Complex instruction pipeline. The accelerator implements composite CISC instructions that combine vector fetching, matrix multiplication, and nonlinear activation into one atomic operation. This microarchitectural fusion eliminates the storage of partial sums in DRAM between network layers, radically reducing inference latency.
10. DRAM (Storage and Byte-addressing of Data)
11. Tensor core and convolutions. The systolic array is configured for direct execution of convolution operations without their software repacking into matrix multiplication. The hardware scheduler decomposes multidimensional windows into an instruction stream, efficiently utilizing MAC units under various stride parameters and filter dilation.
12. Tensor (Multidimensional container for numerical data)
13. Inter-Chip Interconnect interface. For scaling into supercomputer modules, a proprietary Inter-Chip Interconnect is used. The high-speed link with a torus topology ensures direct transfers of gradients and weights between chips without loading the CPU host, creating a logically unified accelerator with total petaflops power.
14. HBM memory management. The memory controller is optimized exclusively for high-bandwidth traffic rather than low latency. The DDR/HBM subsystem operates in batch streaming mode, transferring large blocks of weight coefficients over a wide bus to the activation buffer with minimal overhead on row conflicts.
15. Predication for sparse data. The hardware does not emulate sparsity through post-factum bit masks but uses static packing. Nevertheless, the flow control unit allows deactivating entire segments of the systolic array upon detecting zero activation clusters, avoiding idle multiplication by zero and saving dynamic power.
16. Hardware nonlinearity unit. The matrix block output is directly connected to pipelined functional units computing sigmoid, hyperbolic tangent, and ReLU. The approximation of these functions is performed by piecewise-linear circuits without microcode, ensuring a fixed, deterministic delay of a few cycles per output vector.
17. Batch inference optimization. The architecture is forcibly efficient with batch sizes that are multiples of 8 or 128. The task scheduler groups server requests into epochs to fully utilize pipeline depth, sacrificing the peak latency of a single request in favor of total throughput on the request stream.
18. Ring all-reduce reduction. In Pod topology, TPU chips form a ring accumulator for distributed computations. During synchronous training, the gradient averaging operation is performed without a central aggregation node; data moves in a circle with concurrent algebraic summation in an analog pipeline.
19. Low-level XLA compiler. Computation consumption occurs through a graph compiler performing operator fusion. XLA analyzes the HLO graph and generates binary code, eliminating intermediate tensors by fully unrolling loops and inlining activations in a single pass of the systolic array.
20. Tail truncation mechanism. For variable sequence lengths in recurrent networks, dynamic padding trimming is used. The scheme hardware-ignores cycles associated with virtual zero padding thanks to a precise counter of valid elements in the CISC instruction control word, accelerating NLP model processing.
21. Stochastic rounding. To smooth quantization errors during training, a hardware stochastic rounding function is used. The scheme introduces controlled pseudo-random dithering when truncating the mantissa, transforming systematic gradient bias into white noise, allowing the model to converge under ultra-low precision conditions.
22. Energy-efficient design. The absence of address translation, speculative execution, and cache coherence eliminates hundreds of picojoules per operation. This stripped-down architecture transfers virtually all chip energy directly to MAC logic, delivering a leading TOPS per watt of consumed power.
23. TOPS (Measuring peak performance of AI accelerators)

Comparisons

• TPU vs GPU. Google tensor processors are specifically designed for linear algebra operations, particularly matrix multiplication, which is the foundation of neural network computations. Unlike general-purpose graphics processors, TPUs sacrifice programming flexibility for maximum throughput on low-precision operations such as INT8 and BF16, achieving energy efficiency unattainable for GPUs during inference of large models.
• TPU vs CPU. The central processor is optimized for sequential low-latency computations, complex branching, and managing a wide range of tasks, whereas the TPU is a narrowly specialized matrix coprocessor. When performing a single forward pass of a neural network, the TPU processes data massively in parallel using a systolic array, making it orders of magnitude more performant than a CPU in tensor computation tasks but completely unsuitable for general-purpose algorithms.
• TPU v5p vs TPU v4. Comparing generations within one architecture demonstrates the evolution of inter-chip connection topology: the transition from a toroidal network to a three-dimensional one based on optical switches. The TPU v5p doubles peak floating-point performance, but the key difference is the reduction in latency and increase in throughput when thousands of accelerators interact in a Pod, which is critically important for distributed training of the largest language models.
• TPU vs FPGA. Field-programmable gate arrays provide reconfigurable logic at the hardware level, allowing the implementation of arbitrary data structures and non-standard bit arithmetic. TPUs, in contrast, possess a fixed, rigidly optimized systolic array architecture. Consequently, FPGAs outperform TPUs in tasks with non-standard sparse computations, but trail behind them in clock speed and multiply-accumulate unit density on standard deep learning operations.
• TPU vs NPU. Neural processors from various vendors, like TPUs, are conceptually aimed at accelerating artificial intelligence but fundamentally differ in ecosystem closure. While NPUs in mobile SoCs are oriented toward energy-efficient inference for edge devices with small data packets, TPUs are integrated exclusively into the Google Cloud infrastructure, optimized for HBM memory bandwidth and mass request serving in data centers without access to an open hardware SDK.
• NPU (Specialized processor for parallel tensor computations)

OS and driver support

Interaction with TPUs is implemented not through classic user-space device drivers but through a proprietary library stack, the main one being libtpu. This stack works on top of the standard PCIe driver in the Linux kernel and provides an API for TensorFlow and JAX frameworks, translating their computation graphs into commands for matrix systolic arrays. Environment setup on Google Cloud virtual machines comes down to installing the jax[tpu] or tf-nightly-tpu packages, which automatically load libraries for network interaction with the host machine and low-level HBM memory management.

Security

The security infrastructure is built on hardware chip isolation, whereby the TPU does not store state between user program calls, and all high-speed HBM memory is cleared by a hardware scrubber before allocation to the next client in Cloud TPU multitenant mode. Data encryption during transfer from host to TPU is ensured through an isolated high-speed ICI interconnect network that does not leave the physical cluster boundaries, eliminating the possibility of traffic interception en route to tensor cores. The execution code is isolated within a compiled XLA representation without the ability to execute arbitrary native code on the coprocessor.

Logging

Diagnostic information collection is performed on two levels: on the host side through integration with Google Cloud Monitoring, where the VM agent exports utilization metrics of MXU matrix blocks and memory bandwidth, while on the TPU side, the built-in runtime profiler records the execution trace of each tensor operation in TensorBoard Profile format, allowing the developer to visualize command pipelining by simply inserting the wrapper tf.profiler.experimental.start() into the code and uploading the collected log to a local disk via capture_tpu_profile.

Limitations

The hardware architecture imposes deterministic limitations: tensors must have a static shape and conform to dimension multiples of 128 or 8 for optimal utilization of the MXU systolic array, all operations must fit into the allocated HBM without automatic swapping to host memory, and dynamic control flow at the Python level (if/else branches depending on tensor values) is not translated into the XLA compiler without using specialized JAX primitives like lax.cond, forcing decision-making logic to be moved to the central processor and paying the synchronization delay between host and accelerator.

History and development

Evolution began in 2015 with a closed Google project to accelerate neural networks for ranking, leading to the creation of TPU v1, capable only of inference with INT8 integer precision. Version v2 added training support with bfloat16 computations and unification of chips into pods of hundreds of devices via a toroidal ICI network. v3 introduced liquid cooling and doubled HBM capacity. The Trillium (v6), announced in 2024, features an architecture with 4.7 times higher peak performance per chip compared to v5e, SparseCore support for accelerating embeddings, and a multislice configuration uniting tens of thousands of chips into a single computational domain.