FP16 (16-bit Floating Point) is a half-precision number format that occupies half the memory of standard FP32. It encodes a real number in 16 bits, enabling fast computations at the cost of slightly reduced precision and a narrower dynamic range compared to its 32-bit counterpart.
FP16 is widely used in deep learning to accelerate neural network training and inference on graphics and tensor processors, as it reduces data transfer volumes and power consumption. The format is also in demand in computer graphics for storing HDR images where full precision of color components is unnecessary, and in mobile devices for photo and video processing.
Typical problems stem from the limited range of representable values (up to 65504) and the rapid onset of inaccuracy. Adding numbers of vastly different magnitudes often results in a zero contribution from the smaller addend. Deep neural networks suffer from vanishing gradients when small values round to zero, and exceeding the upper limit produces Not-a-Number (NaN), disrupting the entire computation chain.
How FP16 works
The operating principle is based on distributing 16 bits among three components: one sign bit, five bits for the exponent, and ten bits for the mantissa, according to the IEEE 754-2008 standard. The exponent is stored with a bias of 15, yielding an actual range from –14 to 15, and the value is computed as: (−1)^sign × 2^(exponent−15) × (1.mantissa), except for special cases (zero, denormalized numbers, infinity). Denormalized numbers allow a gradual approach to zero, sacrificing precision to extend the lower boundary.
Unlike BF16 (Brain Floating Point), which inherits 8 exponent bits and only 7 mantissa bits from FP32, FP16 has a wider mantissa range but a significantly narrower dynamic range. This makes FP16 preferable for inference in models sensitive to activation precision, whereas BF16, with its vast exponent range matching FP32, is easier for training as it less frequently requires gradient scaling. Compared to FP32, FP16 dramatically reduces memory bandwidth requirements and enables the use of specialized tensor cores that perform fused multiply-add (FMA) operations an order of magnitude faster; however, numerical stability in complex algorithms demands mixed-precision techniques and loss scaling to keep small values within the representable range of the ten-bit mantissa.
FP16 functionality
- Number representation format. An FP16 number occupies 16 bits and is encoded by three fields: sign (1 bit), exponent (5 bits), and mantissa (10 bits). The exponent is stored with a bias of 15, allowing orders from –14 to +15. This structure provides a dynamic range from
6.10×10⁻⁵to65504. - Exponent calculation and bias. The exponent field is interpreted as an unsigned integer from which a fixed bias of 15 is subtracted. Values
0x00and0x1Fare reserved for denormalized numbers, zero, infinities, and Not-a-Number encoding. All other values form normalized numbers with an implicit leading one. - Normalized values. When the exponent ranges from 1 to 30, the number is interpreted as normalized. The mantissa is supplemented with an implicit integer part of 1, forming the value
1.fraction_bits. This representation maximizes precision in the primary working range and fully complies with the IEEE 754 binary16 standard. - Subnormal numbers and gradual underflow. A zero exponent field with a non-zero mantissa activates subnormal mode. The integer part of the mantissa becomes zero, and the order is fixed at –14. This implements a gradual loss of significant digits, eliminating an abrupt drop-off for small magnitudes at the cost of gradually declining precision.
- Encoding special values. An exponent with all bits set to one (31) signals non-numeric values. With a zero mantissa, this represents signed infinity; with a non-zero mantissa, it is a quiet or signaling NaN. A zero exponent field with a zero mantissa encodes positive or negative zero, preserving sign information.
- Range and precision limits. The maximum normalized value reaches
65504, and the minimum positive normalized value is2⁻¹⁴. Subnormal numbers allow reaching as low as2⁻²⁴. Relative precision is about 3 decimal digits, and machine epsilon equals2⁻¹⁰, defining the minimum distinguishable step for numbers of unit scale. - Default rounding rules. FP16 arithmetic requires rounding a 32-bit exact multiplication result to an 11-bit mantissa. The standard mandates round-to-nearest-even, minimizing systematic bias. Hardware multipliers implement this by adding a rounding bit with sticky-bit correction at the microarchitecture level.
- Conversion from FP32. When reducing precision from standard float, the exponent undergoes overflow and underflow checking. The mantissa is truncated, retaining the upper 10 bits of the explicit part, and the hidden bit is formed by analyzing the FP32 exponent range. If the value exceeds boundaries, infinity is generated, or the subnormal path is triggered.
- Denormalization behavior in the ALU. Upon detecting subnormal operands, the arithmetic logic unit activates microassembler support. Mantissa normalization is performed with dynamic shifting and order correction. This introduces additional pipeline latency; therefore, architectures often optionally support a flush-to-zero mode for acceleration.
- ALU (Performs arithmetic and logical operations)
- Flush-to-Zero mode. Optimization for handling subnormal numbers involves forcing them to zero upon occurrence. The loss of precision in the small-number region is offset by significantly simplified pipeline logic and the elimination of variable delay. This mode is critical for GPU computations where speed determinism outweighs absolute mathematical rigor.
- Half-precision multiply-accumulate. The computational cores of tensor processors perform multiplication of two FP16 matrices with result accumulation in FP32 accumulators. This avoids precision loss in intermediate sums. This function is key in Tensor Cores blocks, where FP16 arrays are multiplied while full precision is preserved for the final addition.
- Tensor (Multidimensional container for numerical data)
- Loss scaling for training. The Loss Scaling technique multiplies the loss function by a large factor before the backward pass. This shifts gradients from the subnormal FP16 zone into the normalized number working range. Before weight updates, inverse division is performed, ensuring that microscopic weight corrections do not vanish into quantization noise.
- Hardware acceleration on GPUs. Modern streaming multiprocessors contain specialized units for FP16 operations with double the throughput of FP32. The vectorization function packs two 16-bit operations into one 32-bit register, doubling the peak theoretical FLOPS performance during mixed-precision matrix multiplication.
- Support in ARM NEON. The AArch64 architecture provides
FMLALandFMLAL2instructions for multiplying FP16 vectors with FP32 accumulation. The core implements bit-width expansion directly in the SIMD unit pipeline, avoiding separate conversion instructions and efficiently packing half-precision operands into 64- and 128-bit registers. - AArch64 (64-bit processor architecture with fixed instruction length)ARM (Energy efficient execution of processor instructions)
- Neural network weight quantization. Model inference uses FP16 for weight storage, halving memory footprint compared to FP32. The deserialization function restores half-precision tensors into multiplier registers, preserving full precision of the input signal within the layer. This enables running large language models on devices with limited video memory bandwidth.
- NaN handling rules. NaN encoding allows diagnostic information to be stored in the mantissa bits. Checking operations set the invalid flag upon detecting a signaling NaN. Quiet NaN propagates through the computation graph, enabling anomaly debugging without abrupt termination; the sign remains indeterminate in most comparison operations.
- Exception flags. The FP16 status block records overflow, underflow, inexact results, and invalid operations. These sticky bits accumulate, enabling a post-analysis function to identify calculation violations after the main simulation or rendering cycle completes without halting the pipeline for interrupt handling.
- Software emulation on CPUs. The absence of native FP16 on x86 prior to the Sapphire Rapids microarchitecture required conversion libraries. The conversion function isolates bit fields via masking, handles special cases through exponent LUT tables, and assembles result bits. Such scalar emulation is an order of magnitude slower than hardware implementations but ensures identical bit-level accuracy.
- x86 (Execution of instructions based on CISC architecture)
- Energy efficiency of computations. An FP16 multiplier consumes significantly less dynamic energy than its 32-bit counterpart. Silicon area shrinks proportionally to the square of the mantissa width. This allows more computational blocks to fit within the same thermal envelope, using the performance scaling function without violating chip thermal limits.
- Impact on optimizer convergence. Adam-like optimizers use FP16 for gradient moments, introducing quantization noise into statistics. For stability, the weight update step remains in FP32, implementing mixed precision. The master weights function preserves a critically important copy of parameters in full precision, synchronizing it with FP16 before the forward pass.
Comparisons
- FP16 vs FP32 (single precision): FP16 uses 16 bits, FP32 uses 32 bits, giving FP32 a significantly larger dynamic range and mantissa precision. FP32 minimizes rounding errors in scientific calculations, while FP16 halves memory and bandwidth requirements, accelerating tensor operations on specialized accelerators at the risk of losing significant digits.
- FP16 vs BF16 (Brain Floating Point): BF16 retains an 8-bit exponent like FP32 but has a 7-bit mantissa versus FP16’s 10-bit mantissa. Thanks to an identical dynamic range to FP32, BF16 is more resistant to overflow during deep network training. FP16 is more precise in representing small quantities but requires gradient scaling to prevent zeroing out due to the narrow exponent range.
- FP16 vs INT8 (8-bit integer): FP16 is a floating-point format supporting a wide value spectrum, whereas INT8 is an integer type with a fixed range. FP16 does not require complex static quantization with calibration, delivering acceptable inference accuracy without scaling factor tuning. INT8 provides double the compute density and energy efficiency for inference but is sensitive to data distribution uniformity.
- FP16 vs TF32 (TensorFloat-32): TF32 internally uses 19 bits, combining FP16’s 10-bit mantissa and FP32’s 8-bit exponent. FP16 is more compact in storage, but TF32 on Tensor Cores achieves FP16 performance without losing the FP32 range. TF32 eliminates the need for manual loss scaling typical of FP16 training, automatically absorbing gradient spikes, which simplifies model mathematical convergence.
- TF32 (Hybrid format for accelerating tensor computations)
- FP16 vs FP8 (8-bit floating point): FP16 provides twice the bits, guaranteeing better mantissa precision and a smoother gradation of magnitudes. FP8 formats
E4M3andE5M2extremely reduce data movement volume and power consumption, but training convergence requires fine-tuning of block quantization schemes. FP16 remains a universal balance solution, whereas FP8 targets the narrow domain of high-performance inference on next-generation accelerators.
OS and driver support
Operating system-level support for FP16 computations is implemented through GPU drivers (NVIDIA CUDA, AMD ROCm) and specialized libraries such as PyTorch or TensorFlow, which translate high-level operations into native half-precision machine code; the driver abstracts the hardware, checking device Compute Capability and permitting FP16 instruction execution only when tensor cores or equivalent blocks are present, and also manages memory allocation with the alignment required for efficient 16-bit format operations.
Data safety
Using FP16 introduces risks of significance loss and overflow; therefore, software implementation of safe computation methods includes automatic loss scaling to prevent small value zeroing during backpropagation, while at the hardware level, exception handling mechanisms control the occurrence of denormalized numbers, NaN, and infinities, allowing either execution termination or substitution of a predefined value by configuring rounding modes and exception masks in the FPU control register.
Logging
Logging of FP16 operations in high-performance systems is performed through computation graph tracing and API call interception by profilers (Nsight Systems, ROCProfiler), which record kernel execution timestamps, data movement volume, and precision switching instances without slowing computations thanks to direct writing to kernel driver ring buffers with subsequent asynchronous disk writing via lightweight tracing protocols.
Hardware limitations
Hardware limitations of FP16 manifest in a reduced dynamic range (5-bit order, 10-bit mantissa), limiting the maximum representable value to approximately 65504 and leading to a quantization step insufficient for precise addition of numbers with differing orders; processors without native support (most x86 cores) emulate FP16 via conversion to FP32, performing operations with double clock latency and double the SIMD unit register consumption.
Evolution
Historically, FP16 emerged as a storage format in industrial graphics (OpenEXR, 2003) for extending pixel dynamic range; then NVIDIA introduced hardware acceleration for operations with FP32 accumulation in the Pascal architecture (2016) for deep learning tasks; subsequently, modern processors (ARM v8.2-A, Apple A11 and newer) and the IEEE 754-2019 standard definitively standardized arithmetic operations, cementing FP16 as a full-fledged computational type with deterministic rounding rules.