What is FP8 (Compact representation of numbers for fast computations)

FP8 (8-bit floating point number) is an ultra-compact numerical format for artificial intelligence. Imagine that instead of bulky 32-bit containers data gets packed into lightweight 8-bit blocks. This radically reduces memory load and accelerates matrix multiplication in neural networks while preserving sufficient accuracy for inference output.

The format has become the de facto standard for inference of large language models (LLaMA, GPT) and diffusion networks for image generation. It is actively used on NVIDIA H100/H200 accelerators (Hopper architecture), Intel Gaudi GPUs, and specialized chips. The main application is deploying models in production where memory bandwidth and minimal response latency are critical, as well as running heavy transformers on edge devices with limited power.

The main difficulty is the reduced dynamic range and the risk of numerical instability. A mantissa of only 2–3 bits poorly handles activations with a large spread of values (outliers), which can lead to loss of significant magnitudes and degradation of model response accuracy. Also, the standard is divided into two subtypes (E4M3 and E5M2), and choosing between them, as well as manually tuning scaling for each network layer, complicates the quantization process compared to the familiar INT8, requiring pipeline adaptation.

How FP8 works

Unlike integer INT8, which operates in linear space with a fixed step, FP8 relies on exponential scaling: bits are divided into sign, exponent (4 or 5 bits), and mantissa. This structure allows covering a colossal range of magnitudes (from microscopic gradients to huge weights) without catastrophic information loss. The hybrid approach with E4M3 for the forward pass (more precise, with dynamic tensor scaling) and E5M2 for backpropagation (wider range) mimics the behavior of FP16/BF16. Next-generation hardware tensor cores perform computations on FP8 twice as fast as on FP16, instantly converting accumulated sums back to higher precision to prevent rounding errors.

FP16 (Compact representation of Floating-Point numbers)BF16 (Floating-point number format for machine learning)

FP8 functionality

FP8 E4M3 format. The first variant of the FP8 format, E4M3, allocates 4 bits for the exponent and 3 bits for the mantissa. This provides a dynamic range optimized for the forward propagation of neural networks, where the precision of weights and activations is critical for maintaining inference quality without significant distribution drift.
FP8 E5M2 format. The second variant, E5M2, uses 5 exponent bits and 2 mantissa bits. The extended dynamic range makes it suitable for storing gradients and accumulators during training, however for inference it is often applied in transformer blocks where extreme activation values occur.
Hybrid scheme mixed precision. At the inference stage, a hybrid tensor encoding scheme is applied: model weights are statically quantized into E4M3 format, and incoming activations are dynamically projected into E4M3 or E5M2. The format choice for activations is determined during the calibration stage to minimize the KL-divergence error of the output logits.
Microscaling of blocks in MXFP8. The MXFP8 microscaling standard groups FP8 elements into blocks of 32 values with a single shared 8-bit E8M0 scaling factor. This emulates the dynamic range of FP16 without the overhead of storing individual exponents, eliminating the problem of overflow and precision loss in deep residual connections.
Hardware acceleration of quantization. Modern Hopper and Ada Lovelace architecture GPUs contain fourth-generation tensor cores. They perform asynchronous conversion of an incoming FP16 tensor to FP8 immediately before matrix multiplication, masking conversion latencies within the compute pipeline and delivering a twofold increase in mathematical throughput.
Block scaling for embedding compensation. Embedding tables are extremely sensitive to approximation. Applying block FP8 with a scale per token subgroup allows keeping the relative cosine distance error below 0.5% with a fourfold memory compression, preserving semantic proximity ranking undistorted.
Calibration of activations in Attention. The attention mechanism generates statistically non-stationary Softmax outliers. Calibration for FP8 uses a strategy of clipping the maximum threshold to a value that ensures a reduction in rounding error variance, which prevents degradation of attention maps on long context windows exceeding 32 thousand tokens.
Quantization of deep FFN layers. Fully connected feed-forward layers are inferentially stable due to the normal distribution of weights. Static quantization in FP8 allows pre-packing weight matrices into texture memory, and during runtime performing a GEMM operation on streaming FP8 vectors, eliminating the unpacking phase and cutting memory bandwidth consumption in half.
GEMM (Matrix multiplication using the row times column method)
Preemptive loading of scales. When dispatching FP8 inference to an accelerator, tensor descriptors contain an out-of-band data stream with scaling factors. The hardware scheduler loads these factors into the multiprocessor register file before the arrival of the main operands, allowing the pipelined tensor core to perform denormalization without pipeline stalls.
Stochastic noise rounding mechanics. To prevent systematic bias in residual networks, stochastic rounding is applied. An LFSR-based pseudo-random sequence generator adds noise to the discarded bits of the FP8 mantissa, probabilistically injecting a carry bit. This preserves the mathematical expectation of the output identical to full precision.
Configuration of FP8 PWM cores. FP8 multipliers within a tensor core function as pulse-width modulators. The product of two 8-bit numbers is internally accumulated in FP16 before transfer to the FP32 summation matrix. This suppresses the accumulation of quantization noise in deep cascades without expanding the bit-width of accumulator registers.
L1 cache management during partitioning. For efficient utilization of Shared Memory bandwidth, FP8 tensors are placed with 128-byte alignment, corresponding to one MX-block cluster. Memory transactions are grouped so that each warp fetches a full scale block and the corresponding data block, avoiding bank conflicts in the L1 cache.
Energy efficiency of the transition. Switching from integer INT8 to floating-point FP8 reduces energy consumption for activation preprocessing by 20% due to the elimination of symmetric requantization operations with a zero point. Direct FP16->FP8 conversion is implemented by combinatorial logic of bit dropping and bias adjustment without integer arithmetic.
Transparent sparsity logic. During FP8 inference, 2:4 structured sparsification is implemented. The hardware block selector in the tensor core reads a bitmask, extracting only the non-zero 2 FP8 elements from each group of four, and passes them to the multiplier. This doubles the effective throughput without changing the logical tensor size.
Unification of RoPE projection scenarios. High-dimensional RoPE positional embeddings require rotation angle precision. The FP8 rotation encoder uses E5M2 for sine-cosine tables and E4M3 for the query and key tensor itself, then FP16 accumulation of complex multiplication is performed to suppress phase noise in score matrices.
KV-cache pipeline optimization. Keys and values are stored in dedicated HBM in E5M2 format, resistant to the shift range of sliding attention windows. When loaded into registers, the dequantization scale factor restores the original FP16, allowing savings of up to 4x in KV-cache memory volume without affecting model perplexity.
HBM (3D stacked memory with silicon vias)
Inference integrity check. The ECC mechanism for FP8 computations is based on residue arithmetic modulo 3. Weight checksums are compared before and after GEMM on the fly, enabling detection of single bit-flip errors in SRAM without costly software duplication of the model run.
SRAM (Fast volatile random storage of bits)ECC (Memory Error Detection and Correction)
Automatic calibration of boundary values. The model profiler at the compilation stage inserts dynamic range observers into the IR graph. The MSE minimization algorithm iteratively selects clipping parameters for each FP8 layer, based on Bayesian optimization over a calibration dataset sample of a hundred images.
nvJitLink software stack. The model translator from ONNX to FP8 inference links separately compiled CUDA kernels for embeddings, attention, and MLP into a single CUBIN. Linking occurs at runtime with just-in-time compilation of scale descriptors, adapting the graph to batch dimensions without recompiling the entire pipeline.
CUDA (Parallel computing on the graphics processing unit)
Determinism of output. FP8 inference guarantees a bitwise reproducible result given identical rounding parameters and reduction order. Unlike BF16, where the non-associativity of addition yields indeterminacy when changing parallelism forms, in FP8 the result is fixed at the quantization stage and does not depend on the number of multiprocessors.

Comparisons

FP8 E4M3 vs FP8 E5M2. The E4M3 format with four exponent bits and three mantissa bits provides higher storage precision for neural network weights due to reduced rounding error. In turn, the E5M2 format with five exponent bits sacrifices mantissa for a wide dynamic range, making it indispensable for storing gradients and activations where surges of large values are likely.
FP8 vs FP16. Switching from 16-bit representation to 8-bit provides a twofold reduction in memory bandwidth and data storage volume requirements. Although FP16 possesses a significantly higher precision margin, FP8 inference practically matches metric quality given hardware support, providing a substantial gain in energy efficiency and batch processing speed on modern accelerators.
FP8 vs INT8. Unlike the INT8 integer format with a uniform quantization step, the floating-point representation distributes discretization levels unevenly. This allows FP8 to adaptively retain high relative accuracy near zero and cover a significant range of signal amplitudes without laborious calibration. As a consequence, adopting FP8 often simplifies the model preparation pipeline, eliminating the need to select scaling coefficients.
FP8 vs FP32. Full 32-bit computation remains the benchmark of accuracy but creates a bottleneck in data bus bandwidth during inference of large language models. Using FP8 reduces memory access time fourfold, critically decreasing latency. Modern mixed-precision methods allow limiting error accumulation, keeping the output activation distribution close to the results of computations in the reference FP32 format.
FP8 vs BF16. The Brain Floating Point 16 format was specifically designed for training neural networks, duplicating the dynamic range of FP32 through seven exponent bits. FP8 radically compresses this bit-width, accelerating matrix multiplications at the cost of increased quantization noise. During inference this forces the use of block scaling and stochastic rounding to compensate for losses, whereas BF16 guarantees stability without auxiliary correction techniques.

OS and driver level support

The implementation of FP8 computations within the inference loop begins with low-level integration into the operating system stack through proprietary libraries and kernel-mode drivers, which abstract the physical features of GPU tensor cores, providing dynamic dispatch of quantization micro-operations; the driver automatically detects the presence of hardware acceleration blocks (NVIDIA Hopper, AMD CDNA3) and replaces standard cuBLAS or rocBLAS calls with optimized convolutions in the hybrid E4M3/E5M2 format, and interaction with user space is carried out through unified memory virtual pages, eliminating weight copying between host and device when changing the data format.

Ensuring computation security

Security when using 8-bit floating-point numbers is provided by hardware isolation of contexts in multi-tenant environments, where the GPU scheduler strictly separates quantized compute streams using the Multi-Instance GPU mechanism, preventing leakage of residual data from shared L2 cache after dequantization operations, as well as by enforcing runtime range validation to block possible attacks through non-numeric values (NaN/Inf) capable of causing denial of service due to avalanche propagation of exceptions in matrix multiplication blocks.

Logging and diagnostics system

The logging infrastructure for FP8 inference is based on instrumentation of the computational graph by the compiler, which injects markers before and after each layer, serializing not only tensor values into tensorboards but also statistics of dynamic range boundary exceedances, while the real-time profiler collects metrics of exponent saturation and the number of denormalized values, writing them to binary logs with minimal overhead through direct access to the cycle performance registers of streaming multiprocessors without stopping the execution pipeline.

Limitations

The fundamental limitation of FP8 lies in the catastrophic loss of precision when tensor activations exceed the representable maximum (448 for E4M3), which forces the implementation of delayed scaling techniques with carry-over of multipliers to subsequent operations; however, this does not prevent the accumulation of rounding error in deep residual networks, where the small three-bit mantissa provokes a systematic bias when summing millions of gradient updates, requiring a hybrid switch to BF16 for critical attention layers.

Evolution

The history of FP8 development represents a transition from emulation on integer ALUs in scientific simulators to hardware implementation in the Hopper microarchitecture, made possible by the joint work of a consortium of manufacturers on the IEEE P3109 standard, which legitimized two interchangeable encodings for forward and backward passes, and the final adaptation by compilers of automatic mixed-precision quantization mechanisms, turning point chip optimization into a universal deployment pipeline for large language models.