XGMI (AMD high-speed coherent Inter-Die bus)

XGMI is a direct physical interface that unites multiple Graphics Complex Dies (GCDs) into a single logical processor. Simply put, XGMI makes the system see a bundle of different chips as one monolithic graphics card with shared memory, hiding the boundaries between dies from the driver and applications.

The technology is used in professional AMD Instinct accelerators of the MI200, MI300 series and above. There, XGMI connects compute dies directly to each other under the common package lid, providing coherent access to a unified pool of HBM memory. In server solutions, Infinity Fabric bridges based on XGMI also link multiple physical accelerators into a single network for direct data exchange without CPU involvement.

The main problem is extreme sensitivity to mechanical deformation and the quality of the contact pad. The slightest poor solder joint, dust on contacts, or a microcrack in the PCB under the bridge leads to a complete loss of connection between dies. This causes initialization errors, freezes during driver loading, or one of the chips dropping out of the topology, often misdiagnosed as a failure of the memory chip itself.

How XGMI works

The operating principle is based on extending the on-die coherent Infinity Fabric beyond a single die. Unlike simple SLI/CrossFire, where frames are rendered alternately and data is duplicated in each accelerator’s memory, XGMI transmits coherency protocol traffic, making the memory controllers of all dies work as a single multi-channel array. Bandwidth reaches hundreds of gigabytes per second per link. Unlike NVLink bridges, where exchange often requires explicit copying through CUDA cores, XGMI allows compute die units to directly and atomically access physically remote memory addresses of another chip without initiating DMA procedures.

Compared to the classic PCI Express bus, which is a non-coherent packet transport with descriptor overhead, XGMI appears as an extension of the chip’s own internal router: read requests and cache line invalidations pass directly, bypassing the stage of translation into universal transport packets. Physically, this is implemented through arrays of miniature differential pairs in an organic interposer or through direct Embedded Multi-Die Interconnect Bridges, soldered to tiny bumps on the die. This approach makes the latency to a remote memory bank almost indistinguishable from the latency to a local one, which is critically important for computational tasks with irregular data access.

XGMI functionality

1. Infinity Fabric bus for inter-die connections. XGMI technology (Inter-die GPU Link) is a proprietary implementation of the AMD Infinity Fabric bus, adapted for high-speed communication between Graphics Complex Dies within a single physical package or module.
2. Physical layer and topology. The connection is based on differential signaling using SerDes transceivers optimized for ultra-short trace distances on a common silicon substrate or interposer, forming bidirectional point-to-point links between neighboring dies.
3. Memory coherency protocol. XGMI extends the standard Infinity Architecture protocol for GPUs, providing GMI3 coherency semantics, where agents can directly and atomically operate on data in the local caches and memory of a remote die without explicit synchronization instructions.
4. Protocol stack architecture. The functional stack is divided into a transaction layer, managing read/write and snoop requests, and a link layer, responsible for virtual channels (VC) that prevent request and response traffic deadlocks, packing into flow control units (FLITs), and hardware error correction (ECC).
5. ECC (Memory Error Detection and Correction)
6. Unified address space. All dies are logically combined into a single physical memory pool. Memory controllers on different dies operate in interleaving mode, transparently mapping physical address space pages through XGMI inter-domain links without data copying.
7. Low-Order Interleaving mode. To minimize latency on large assemblies, interleaving is applied over small granular blocks (256 bytes or more). This distributes sequential accesses from compute units across all available HBM controllers simultaneously, balancing the bus load.
8. Split transaction support. The protocol implements a mechanism for splitting address and data phases (split transaction). The master device can send a request packet and release the channel without waiting for the cycle to complete, which is critically important for hiding the delays of long transitions between dies.
9. Virtual Channels QoS. To isolate latency-sensitive control traffic from bulk DMA data streams, the physical link is logically segmented. Command traffic, coherency requests, responses, and data arrays are assigned to different virtual channels with separate buffers.
10. Bandwidth scaling. A single XGMI link in current CDNA architectures provides speeds up to 100–150 GB/s in each direction. By combining multiple physical links between a pair of dies, an aggregated bandwidth of over 1 TB/s per module is achieved.
11. Hardware routing subsystem. In configurations with four or eight dies, a distributed routing fabric is activated. Transactions pass through intermediate dies in transit, using routes with the minimum number of hops according to a static table programmed during initialization.
12. Deadlock prevention mechanism. The XGMI fabric uses algorithms based on graph theory and duplicate virtual networks (dateline routing). Traffic crossing a conditional turn line in the route is automatically transferred to a separate buffer pool, eliminating cyclic dependencies.
13. Snoop Filter. To save bus bandwidth, each die is equipped with a distributed snoop filter. It tracks cache line tags belonging to remote compute units and filters out unnecessary broadcast invalidation requests at the local level.
14. Link power management. The physical layer supports dynamic scaling of bus width (from x16 to x1) and SerDes clock frequency depending on channel utilization. During idle periods, the link can enter a low-power L1 state with clock signal transmission disabled.
15. Integration with HBM subsystem. The XGMI controller is closely linked to HBM memory clients through a common die crossbar. A request from a remote compute unit can pass through the XGMI fabric and seamlessly enter the physical channel of the memory stack, bypassing local caches.
16. Remote atomic processing. The bus supports completing atomic arithmetic-logical operations directly in the remote die’s memory controller. This eliminates the need to fetch the cache line with data locally, modify it, and send it back, radically reducing traffic.
17. Data integrity on the line. In addition to standard ECC on FLIT packet bits, XGMI implements end-to-end referential integrity protection. Transactions are accompanied by check codes, guaranteeing that the memory address or write contents were not corrupted during transmission between dies.
18. Write order synchronization. To comply with GPU memory models, the link protocol contains a Store-Opening Barrier mechanism. Write acknowledgement packets from remote memory are used for global ordering of data visibility operations within the synchronization domain.
19. Fabric error isolation. When uncorrectable CRC errors occur, the XGMI controller initiates a poisoned data packet procedure, preserving the infected bit up to the application level without triggering an immediate halt, while the hardware replay buffer ensures transparent retry of the transaction.
20. Configuration registers. The link topology is set at boot time by System BIOS or the driver through protected SMU registers. The operating mode (xGMI vs xGMI-AID) and the address interleaving map are configured before the bus exits the cold reset state to ensure node symmetry during OS initialization.
21. Monolithic device emulation. The ROCm driver abstracts the complex XGMI topology, creating the appearance of a single logical device. All remote die memory pools are declared as local NUMA nodes, allowing the runtime API to transparently migrate data and queue tasks.

Comparisons

• XGMI vs NVLink Bridge. XGMI is a SerDes-based physical interface for directly connecting AMD GPU dies, whereas NVIDIA NVLink bridges historically served as an external card-to-card channel. XGMI provides a coherent memory fabric inside the module, creating a single address space without software-visible gaps, which is critically important for seamless computation scaling in multi-die Instinct MI-series packages.
• XGMI vs NVSwitch. In AMD architectures, the XGMI topology is often built on an all-to-all principle directly, whereas NVSwitch acts as a centralized switch. AMD’s solution reduces latency and eliminates a separate intermediary device with a small number of nodes in a rack, while NVSwitch provides superior bandwidth when aggregating eight or more GPUs, sacrificing physical layer simplicity for total non-blocking performance.
• XGMI vs Infinity Fabric (Inter-Socket). The on-die XGMI implementation is functionally identical to external Infinity Fabric links but optimized for ultra-short distances. The main difference lies in energy efficiency: XGMI operates at the level of inter-silicon bridges with reduced signal voltage, while the inter-socket version of Fabric is designed for greater path attenuation. The software coherency model, however, is fully unified.
• XGMI vs CXL (Compute Express Link). XGMI implements a private AMD coherency protocol, natively integrating graphics memory as a single pool, while CXL is an industry standard based on PCIe 5.0/6.0 physics. XGMI’s priority is ultra-low latency direct access to video memory frames within a closed ecosystem; CXL’s advantage is openness and suitability for heterogeneous systems, including shared memory pools with central processors.
• XGMI vs UALink (Ultra Accelerator Link). The comparison is prognostic in nature: XGMI is a working current-generation technology, while UALink is being developed by an industry consortium for open scaling of inference and training. The difference lies in the vector of development: XGMI is tailored for vertical integration of AMD accelerators, while UALink aims to create a universal transport with simple read/write semantics, potentially threatening the monopoly of NVLink in the high-performance switched network segment.

OS and driver support

The XGMI architecture is implemented at the firmware (AGESA) and low-level library level, so the operating system does not manage the physical transport of inter-die connections directly. The driver (amdgpu) initializes the topology through xGMI IP blocks at kernel module load time, reading distances between GPUs from ACPI tables (SLIT/HMAT), after which it activates software tunnels (KFD for compute, DRM for display) over hardware links, creating a single logical device without explicit OS scheduler intervention in packet routing.

Security

Data transmission between dies is protected by hardware scrambling and integrity control at the physical layer (PHY), preventing passive signal interception during substrate probing. The xGMI controller within the Data Fabric applies request identifier virtualization, isolating the streams of different virtual machines in SR-IOV environments, and for memory pages shared across inter-socket space, an access rights verification mechanism is used on the System Request Interface side, blocking unauthorized transactions before they enter the remote coherency directory.

Logging

The xGMI subsystem does not keep text logs independently but exports hardware event counters (correctable and uncorrectable CRC errors, loss of sync, packet retries) through the RAS interface to the Linux kernel ELOG subsystem or processor MCA banks. Upon detecting a critical failure, such as lane degradation, the controller generates a Machine Check Exception, and the amdgpu driver records the detailed failure context in the system log via the standard EDAC mechanism, including the affected die identifier and lane number.

Limitations

The main design limitation of XGMI is the mandatory synchronous clock grid for all assembled dies, which excludes hot-plugging or asynchronous frequency scaling within a single domain. The topology supports only fully connected or daisy-chain configurations without dynamic on-the-fly route reconfiguration, and the maximum conductor length on the interposer or bridge chiplet is strictly limited by signal attenuation, making it impossible to extend the bus beyond a single packaged module without using active retimers with substantial added latency.

History and development

The interconnect evolved from the proprietary Fiji bus (first generation HBM integration) into a standardized physical layer Infinity Fabric for discrete GPUs starting with Vega 10, where the term xGMI first appeared as a coherent access tunnel to the memory of a neighboring accelerator. In the CDNA architecture (Arcturus, Aldebaran), the interface was expanded to eight links per die with unified addressing support, and by the MI300 generation, the transport layer is implemented directly on a 3D packaging basis with hybrid copper pillar bonding, where the xGMI protocol functions as a full mesh network between compute chiplets with bandwidth exceeding 896 GB/s per direction and direct hardware CPU-GPU coherency via Infinity Fabric 4.0.