HEVC (H.265) is a video compression algorithm, the successor to AVC (H.264). It was created to deliver high-quality images (up to 8K) at half the bitrate of its predecessor. Simply put, files take up less disk space and streams require a narrower internet channel without a visible loss of picture clarity.
The main application area is streaming. HEVC is the foundation of 4K content on Netflix, YouTube, and DVB-T2 satellite television. Professional video cameras and drones record heavy HDR footage in this codec. The technology is also indispensable in surveillance systems and video conferencing, as it allows archiving days of recordings without purchasing huge servers.
The main drawback is the demand for computational resources. Old laptops and smartphones without hardware decoding overheat and drain the battery quickly. The complex licensing system of HEVC Advance and MPEG LA long hindered the codec’s adoption in browsers, causing part of the web to switch directly to the open AV1. Users also often cannot open an HEVC file on a PC without installing a paid extension from the Microsoft store.
How HEVC works
The codec analyzes the image much more deeply than the outdated AVC. While the previous standard divided a frame into macroblocks strictly 16×16 pixels, HEVC introduces a flexible structure — coding tree units (CTU) up to 64×64 pixels in size. This allows outlining large homogeneous areas (sky, wall) more precisely without fragmenting them into many small pieces, saving bitrate.
Inside this tree, improved intra-prediction is triggered: instead of 8 directions like its predecessor, 33 or 35 motion vectors are analyzed, providing smooth color transitions without artifacts. Inter-frame compression is supplemented with motion compensation technology accurate to a quarter of the luma pixel and one-eighth of the chroma pixel, as well as the use of multiple reference frames simultaneously. The final stage is adaptive filtering. Instead of simple deblocking, Sample Adaptive Offset (SAO) is activated, which corrects ringing and blurring at object boundaries on the fly, making the picture visually cleaner with the same mathematical losses.
HEVC functionality
- Block coding with a tree structure. Frame partitioning is performed into coding units (CU) up to 64×64 pixels. Unlike H.264 macroblocks, a flexible quad-tree structure is used here, allowing adaptive selection of block size based on image content to enhance compression efficiency.
- CU (Managing the execution of machine instructions of the processor)
- Partitioning into prediction blocks. A coding unit is recursively divided into prediction units (PU). Asymmetric partitioning modes are supported, where one rectangle can be larger than another. This allows describing the boundaries of moving objects more accurately, minimizing prediction error and the bitrate of motion vector transmission.
- Partitioning into transform blocks. The residual signal is processed by transform units (TU), formed according to a residual quad-tree structure. TU size can vary from 4×4 to 32×32. The transform matrix is applied independently to each TU, and the quad-tree adapts the transform size to the local spectral characteristics of the signal.
- Discrete cosine transform (DCT) of increased dimensionality. In addition to the basic DCT matrices, transform matrices of sizes 8×8, 16×16, and 32×32 are used. The integer approximation of DCT is performed using partially factorized matrices, ensuring high computational accuracy with fixed intermediate data bit depth to reduce the accumulation of rounding errors.
- Discrete sine transform (DST) for intra-frames. For 4×4 residual blocks obtained after intra-prediction of luma, integer DST is applied. Statistically, the prediction error amplitude increases towards the block boundaries, and the DST basis functions provide better energy compaction of such a signal compared to traditional DCT.
- Extended set of intra-prediction directions. The intra-prediction mechanism supports 35 modes for luma blocks, including planar (surface) and DC (average), versus 9 in AVC. The high angular density of directions (with an increment step of 1/32) drastically reduces residual energy in textured areas, improving the subjective reconstruction quality.
- Planar and DC prediction. The planar mode generates a smooth gradient surface based on four corner samples, effectively coding smooth brightness changes. The DC mode fills the block with the averaged value of neighboring pixels, which is optimal for homogeneous areas. Both algorithms minimize visible blocking artifacts in flat regions.
- Chroma component prediction. Horizontal, vertical, planar, DC modes are available for chroma blocks, as well as borrowing the luma component direction (DM mode). The correlation of textural boundaries in different color planes is used to inherit the prediction angle, increasing the accuracy of cross-component restoration without additional vector signaling.
- Smoothing reference sample filtering. Before intra-prediction of large blocks, an adaptive low-pass filter is applied to boundary reference pixels. The procedure suppresses speckle noise on reference lines, preventing artifact propagation and improving the homogeneity of the predicted block without losing directional sharpness on strong edges.
- Transform skip coding mode. With active Transform Skip for 4×4 blocks, quantization is applied directly to spatial residual values, bypassing the DCT/DST stage. This improves the decoding quality of synthetic content, screencasts, text, and animation with sharp transitions, where frequency decorrelation generates undesirable Gibbs ringing.
- Extended set of motion vectors. Inter-frame prediction uses Advanced Motion Vector Prediction (AMVP) technology. Two candidate lists are formed from spatial and temporal neighbors, allowing flexible selection of the most probable motion vector. Differential coding transmits only the index in the list and the delta, radically reducing bit costs.
- Vector (Ordered storage of numbers in continuous memory)
- Merge mode. Merge mode transmits exclusively the donor candidate index, fully inheriting all its motion parameters without signaling vectors and reference indices. This approach is extremely efficient in static backgrounds and large objects moving translationally, where neighboring blocks have an identical motion model.
- Sub-pixel precision interpolation. For the luma component, accuracy reaches 1/4 pixel, for chroma — 1/8. Interpolation uses separable 8-tap FIR filters for half-pixels and 7-tap for quarter-pixel positions. The high filter cutoff slope prevents damping of high frequencies during fractional shift compensation.
- Improved deblocking filter. The H.265 filter operates inside the coding loop on an 8×8 grid, having lower computational latency. The cascading dependency of operations is reduced, and parallel processing of rows is possible. Decisions on smoothing strength are made based on the analysis of artifacts caused by quantization, not real object contours.
- Sample adaptive offset (SAO) filter. SAO classifies reconstructed pixels by intensity or edge properties and applies additive correction. Specific artifacts are suppressed: banding in smooth gradients and ringing at contrast boundaries, achieved by adding table offset values calculated by the encoder at the CTU level.
- Parameter signaling dependency mode. The bitstream uses high-level syntax based on the adaptation parameter set (APS). This allows reusing identical quantization tables and SAO parameters between slices and frames without duplication. The reference architecture saves bitrate by eliminating redundant header overhead.
- Parallel processing with tiles. The frame is divided by a regular grid into rectangular regions, coded and decoded independently. Tiles do not use each other’s data in prediction and entropy modeling, breaking wavefront synchronization. This ensures deterministic parallelism on multi-core architectures at the cost of a slight compression loss.
- Wavefront parallel processing (WPP). Wavefront Parallel Processing provides parallelism at the LCU row level. The encoder starts processing the next row as soon as the first two blocks of the previous row are completed. The restored CABAC state is passed on in a relay fashion, preserving the vertical dependencies of the context model, unlike the rigid isolation of tiles.
- CABAC entropy coding. A single context-adaptive binary arithmetic coding stream is used. The context update mechanism has been redesigned: the total number of models is reduced, and sensitivity to rapid statistical fluctuations is lowered. The peak bin decoding throughput has significantly increased due to the reduction of data dependencies.
- Transform coefficient coding. A sign data hiding mechanism is introduced. When the sum of absolute non-zero coefficients in a 4×4 CG is even, the sign of the last element is not transmitted but deduced by the decoder based on parity. Adaptive scanning is applied, and the last significant coefficient is explicitly coded with a coordinate pair, optimizing the traversal of sparse significance maps.
Comparisons
- HEVC vs AVC (H.264). HEVC provides approximately a twofold reduction in bitrate while maintaining identical subjective image quality through the use of coding units up to 64×64 pixels and more advanced motion vector prediction, whereas AVC is limited to 16×16 macroblocks, making it less efficient for 4K resolution.
- HEVC vs VP9. Unlike the open VP9 codec, HEVC offers slightly higher compression efficiency at low bitrates thanks to improved deblocking and the SAO filter; however, VP9 remains free to implement, eliminating the complex royalty system characteristic of H.265 content distribution.
- HEVC vs AV1. AV1 technologically surpasses HEVC, achieving bitrate savings of up to 30% due to a greater number of intra-predictions and affine transforms, but at the cost of a manifold increase in encoding computational complexity, making HEVC a more practical solution for real-time encoding tasks.
- HEVC vs VVC (H.266). VVC extends the legacy of HEVC by introducing a quad-tree with multi-type splitting and a recurrent neural network post-filter, yielding a compression gain of about 40%; however, HEVC hardware support is now ubiquitous, while mass decoding of H.266 requires resources available only in the latest SoCs.
- HEVC vs HEIF (container format). The fundamental difference is that HEVC is a video stream compression method, while HEIF is a structural container for storing images that can use HEVC algorithms to encode single frames, providing superiority over JPEG through support for HDR and 10-bit color depth.
OS and driver support
Hardware acceleration of HEVC is implemented through operating system software interfaces, such as DirectX Video Acceleration (DXVA) and Direct3D 12 Video in Windows, Video Toolbox in macOS and iOS, as well as Video4Linux (V4L2) with requests to KMS/DRM in Linux, where the desktop loads open-source Mesa drivers by default (for example, for VA-API); closed-source drivers from NVIDIA and AMD require the installation of proprietary libraries, while in Windows the universal HEVC profile (Main 10) is supported by built-in WDDM drivers starting from version 2.0; however, to activate the codec in players, the user often needs to install an official extension from the Microsoft Store or activate the hardware scheme through Intel Quick Sync and NVDEC interfaces, which transfer the video stream to a specialized ASIC block bypassing the central processor.
Security
The security of the HEVC video pipeline is based on integration with hardware enclaves and digital rights management schemes, where video frames are decoded in an isolated protected memory area (Protected Media Path or Trusted Execution Environment) inaccessible for screenshot capture at the operating system kernel level, and external video buses are encrypted using the HDCP 2.2 protocol during transmission to the display, preventing interception of the uncompressed digital stream using physical analyzers; the HEVC bitstream itself can contain SEI messages with license revocation parameters, and frames loaded into the encoder undergo integrity checks via checksums, eliminating injections of malicious data into frame buffers through slice parser vulnerabilities.
Logging
HEVC encoding and decoding systems use multi-vector logging, where the low-level driver records bitstream checksum reset statistics and frame error states into the system trace, and the application itself captures extended metadata with temporal level identifiers (Temporal ID), the active slice type (I, P, B), and the number of bits spent on signaling the coding tree unit partitioning in each processed fragment; when debug mode is enabled, the encoder outputs decisions on motion vector selection and intra-prediction modes into a structured text file synchronized with the frame number, allowing engineers to build bitrate distribution heat maps and identify anomalies in the hypothetical reference decoder (HRD) buffer fullness at the Coded Picture Buffer (CPB) syntax level.
Limitations
The main technical limitation of HEVC lies in patent fragmentation and licensing barriers, forcing chip manufacturers to implement Main 10 profile support only in premium system-on-chip models, while budget devices can only hardware decode the 8-bit Main profile, preventing correct HDR content playback; in addition, the maximum coding tree unit size of 64×64 pixels and the mandatory support of 35 intra-prediction modes create significant computational complexity, ruling out purely software decoding of ultra-high resolutions on low-power embedded processors without a built-in DSP coprocessor, and strict rules for positioning reference frames in the RefPicList0 and RefPicList1 lists limit prediction flexibility in scenarios with abrupt content changes.
History and development
HEVC standardization was completed in April 2013 as a joint project of the ISO/IEC MPEG and ITU-T VCEG expert groups, who joined forces within the Joint Collaborative Team on Video Coding (JCT-VC) to replace the obsolete H.264 with the goal of halving the bitrate at equal visual quality by introducing a hierarchical quad-tree structure for coding unit partitioning and an improved deblocking filter with subsequent sample adaptive offset (SAO), which enabled efficient processing of formats up to 8K; however, the fragmentation of patent pools (MPEG LA, HEVC Advance, Velos Media) stimulated the industry to move towards developing an open alternative in the form of AOMedia Video 1 (AV1), continuing the evolution of HEVC itself through extensions such as Scalable HEVC (SHVC) for spatial scalability and 3D-HEVC for multi-view stereoscopic video.