Peer2Peer (Direct memory access between GPUs)

Peer2Peer is a technology that allows one graphics processor to directly read or write data to the video memory of another GPU without going through the central processor or system RAM. This reduces latency and offloads the PCI Express bus, creating a high-speed bridge between accelerators.

The technology is in demand in multi-card systems for training large neural networks where the model does not fit in the memory of a single accelerator. The NVIDIA NCCL library actively uses P2P for collective operations, and frameworks like PyTorch use it for fast gradient synchronization. It is indispensable in scientific modeling and tasks requiring the exchange of large volumes of data between GPUs in real time.

The main problem is topology dependence. If devices are connected to different processor sockets or through a PCI switch, direct access is blocked. Input-output address translation (IOMMU) can prohibit P2P for security reasons. Memory barrier difficulties also arise: the absence of explicit synchronization can lead to reading stale data, since GPUDirect does not guarantee atomicity of operations between different devices.

IOMMU (Isolation of direct memory access addresses)

How Peer2Peer works

The operating principle is based on a unified virtual address space and direct memory mapping (BAR). Unlike classic copying through the CPU, where the buffer sequentially passes through the host RAM, here the driver configures page tables so that the physical addresses of the target GPU are visible to the initiator. Essentially, reading from the memory of a neighboring accelerator looks like a local transaction. The mechanism relies on the PCI Express extension — Address Translation Services (ATS), allowing devices to exchange address translations without root complex involvement. Compared to GPUDirect RDMA, which requires a network adapter to go beyond the node, P2P works exclusively within a single machine via the PCIe bus.

Compared to standard copying through Unified Memory, where pages migrate slowly causing page faults, P2P provides access without moving data, keeping it in the owner’s local memory. The key difference from NVLink is the physical layer: NVLink provides higher bandwidth for inter-node connections, while P2P via PCIe is implemented in software and hardware on top of the standard protocol, ensuring cross-platform compatibility and interoperability with a wide range of hardware.

Peer2Peer functionality

  1. Direct GPU memory access. Peer-to-Peer Memory Access technology allows two graphics processors to directly access each other’s local video memory without central processor and system RAM involvement. Data is transferred over the PCI Express bus, bypassing intermediate copying into the host buffer, which dramatically reduces latency.
  2. Routing through a PCIe switch. The physical possibility of direct access is determined by bus topology. For successful P2P activation, both accelerators must be located behind a single PCIe switch supporting ACS (Access Control Services) in transparent mode or disabled. Passing through the CPU root complex blocks direct data transfer.
  3. Virtual address space. The CUDA driver unifies addressing by projecting the third-party device’s local memory into the common virtual space of the current context. A pointer to GPU 1 memory becomes valid in a kernel executing on GPU 0. Address translation occurs at the hardware level on the chip side through GMMU page tables.
  4. CUDA (Parallel computing on the graphics processing unit)
  5. UVA coherence mode. The technology is the basis for Unified Virtual Addressing. In combination with P2P, the program gets a single pool of pointers without distinguishing the physical location of data. The kernel can perform atomic addition or comparison operations in adjacent GPU memory, enabling synchronization without copying data back to the host.
  6. Direct kernel launch (P2P Kernel Launch). CUDA allows launching a kernel on GPU 1 by passing arguments directly from GPU 0 memory. This technique eliminates CPU launch, which is critically important for MPI-like paradigms where the controlling accelerator independently distributes computational tasks among neighboring cards.
  7. Software-level activation. Enabling access requires an explicit call to cudaDeviceEnablePeerAccess for a specific pair of devices after checking the cudaDeviceCanAccessPeer flag. The call is expensive and is usually performed once during the initialization phase, establishing a unidirectional communication channel between two contexts.
  8. Memory unification with migration. In the CUDA 6+ era, P2P technology is integrated into Managed Memory. When a kernel accesses a page physically located on another GPU, the hardware migration mechanism generates a page fault and copies data over the P2P channel without interrupting thread execution. This allows overlapping communication and computation.
  9. RDMA support via GPUDirect. P2P is the foundation for GPUDirect RDMA. InfiniBand or RoCE network adapters, using the internal PCIe bus, are capable of directly reading data from GPU memory while CUDA kernels of another GPU write to the same place. This implements end-to-end data transfer without host synchronization.
  10. Atomic operations between GPUs. Hardware support for atomics over the PCIe bus is a key feature. A device can execute atomicAdd on a pointer referencing an adjacent card’s buffer. This enables the implementation of lightweight distributed mutexes and memory-level barriers without using expensive system calls.
  11. Bandwidth multiplier optimization. Despite the physical bus limit, P2P duplex mode theoretically doubles the aggregated bandwidth in a two-card topology. Using simultaneous cudaMemcpyPeer copies in both directions allows saturating upstream and downstream PCIe channels simultaneously without collisions.
  12. Buffering and queue depth. The efficiency of P2P transfers critically depends on saturating the command pipeline. Sending small data fragments generates TLP DMA transaction setup overhead. Aggregating data into large blocks up to 4 MB and using asynchronous streams allows hiding DMA engine initialization latency.
  13. NVIDIA NVLink and NVSwitch. In architectures starting with Pascal and Volta, NVLink replaces PCIe as the physical transport for P2P, increasing bandwidth up to 900 GB/s in the DGX complex with NVSwitch. The CUDA runtime abstracts the transport layer, allowing identical P2P API usage for NVLink and PCIe without code modification.
  14. Synchronization via IPC descriptors. Inter-Process Communication is used for data exchange between different processes. P2P access in such a scenario exports memory via cudaIpcGetMemHandle. The receiving process opens the descriptor, obtaining a direct pointer to the sender’s address space, bypassing the read/write system call.
  15. IPC (Instructions per cycle)
  16. Memory barriers and data visibility. Hardware does not guarantee end-to-end write ordering without explicit primitives. After writing to a P2P region, the source GPU must execute a __threadfence_system memory fence operation. Only this guarantees that the data has physically left the chip caches and become visible to the target device via the bus.
  17. Diagnostics via nvidia-smi topo. The utility allows building a topology matrix where links with NV value indicate single-node connection via NVLink, and PIX indicates P2P availability through a single PCIe switch. Links labeled SYS require traversal through the processor’s QPI/UPI, making direct memory access impossible or extremely slow.
  18. BAR1 limitation bypass. Direct access depends on the size of the BAR1 (Base Address Register) window mapped into the address space. If the window size is insufficient, the driver is forced to use intermediate copying to system memory. For large memory capacities such as 80 GB HBM, it is critically important that the firmware reserves a BAR1 of sufficient size.
  19. HBM (3D stacked memory with silicon vias)
  20. P2P mode with MIG partitions. In the Ampere architecture and later, direct access support is implemented for individual MIG instances within a single physical GPU, as well as between MIG partitions of different chips. This allows isolated inference servers to exchange computation graphs at high speed without CPU-gateway emulation.
  21. Transaction error control. Unlike standard copying, P2P transactions are sensitive to ECC errors on the PCIe bus. If an uncorrectable data integrity error occurs during a transaction, the CUDA driver terminates the context since the chipset mechanism does not allow isolated restart of a specific DMA transfer.
  22. ECC (Memory Error Detection and Correction)
  23. Security and access control. The IOMMU hardware block (VT-d/AMD-Vi) can block pass-through GPU access to foreign memory if ACS is forcibly enabled. In configurations with address virtualization, P2P translation must go through the same tables as other DMA requests, preventing reading another virtual machine’s memory.
  24. AMD-Vi (Hardware I/O virtualization)
  25. Non-blocking completion polling. To minimize computational unit idle time, callbacks based on cudaStream are used. Upon P2P copy completion, an event is generated capable of directly launching a new handler kernel, bypassing the CPU wait cycle. This implements a data processing pipeline fully resident in a heterogeneous GPU cluster.

Comparisons

  • Peer2Peer vs RDMA. Peer2Peer technology provides direct interaction between GPUs within a single node via the PCI Express bus, whereas RDMA is oriented toward inter-network data exchange between remote hosts, bypassing the recipient’s central processor. The main difference lies in topology: P2P minimizes latency inside the server using bus hierarchy, while RDMA scales transmission over cluster distances via InfiniBand or RoCE, requiring network adapters.
  • Peer2Peer vs GPUDirect Storage. The Peer2Peer function organizes data flow between computational accelerators, eliminating copying into host system memory, which is critical for multi-node calculations. In contrast, GPUDirect Storage creates a direct I/O channel between GPU and NVMe solid-state drives. If P2P accelerates the exchange of parallel computation results, then GDS optimizes loading massive datasets directly into accelerator memory, bypassing the CPU.
  • Peer2Peer vs Unified Memory. The Peer2Peer direct access mechanism is distinguished by hardware traffic routing through PCIe switches without driver involvement in the data transfer critical path. Unified Memory technology, on the contrary, creates a virtual address space with software page migration. P2P does not require complex page fault handling, providing deterministic bandwidth, while Unified Memory sacrifices performance for programmer-transparent access to the entire data volume.
  • Peer2Peer vs NVLink. The Peer2Peer interface is implemented on top of the standard PCI Express hierarchy and is often limited by inter-processor link bandwidth. The NVLink direct connection forms a dedicated mesh channel with significantly higher data transfer speed and automatic cache coherence capability without address translation through the host. The key difference is that NVLink is a physical medium for building tight computational groups, while P2P is a logical protocol on a universal bus.
  • Peer2Peer vs CPU Memcpy. Using Peer2Peer eliminates the classic copying scheme through a central processor intermediate buffer (Staging Buffer), where data descends from one GPU to RAM and only then ascends to the second. Compared to the Memcpy function, which uses chipset resources, P2P closes traffic within the PCI Express subsystem, reducing the load on processor memory controllers and significantly decreasing transfer latency without a transit stop on the host side.

Driver and OS support

Implementation of direct access between video accelerator memory requires loading the nvidia-peermem kernel module, which creates a character device for registering GPU BAR spaces in the operating system memory subsystem, as well as activating the nvidia-fabricmanager service for managing the GPU cluster on NVSwitch server platforms — the driver performs manual programming of PCIe device page tables through base address registers and configures request routing in the root complex, bypassing the central processor.

Security

The protection mechanism is built on forced isolation of virtual address spaces through IOMMU (Intel VT-d and AMD-Vi), where each P2P transaction is checked by the hardware address translation block for compliance with device context tables, and the driver additionally validates that both GPUs belong to the same isolation domain and a common PCIe switch, prohibiting pass-through access when crossing root port boundaries or with active hardware virtualization with device passthrough.

Intel VT-d (Hardware isolation of direct device access)

Logging

Activity diagnostics is implemented through the Linux kernel tracing mechanism — ftrace events in the p2pmem subsystem record page mapping success, coherence synchronization errors, and transaction timeouts, while the NVIDIA driver exports detailed direct access operation counters to the nvidia-smi interface (P2P Throughput and BAR1 Usage fields), allowing real-time observation of data volume bypassing system memory and identification of link degradation.

Hardware and topological limitations

Functionality is available exclusively on accelerators supporting PCI Express Base Specification 3.0 and higher with mandatory placement of both devices behind a single PCIe switch without crossing the Root Complex, and the manufacturer hardware-blocks P2P for consumer-grade GeForce cards at the video BIOS subsystem level, leaving the possibility only for professional Tesla and Quadro solutions, while the maximum volume of simultaneously mapped BAR1 memory is limited by the physical translation window registers.

Architecture evolution

The initial implementation appeared in NVIDIA driver version 346 for CUDA 6.5 as an experimental Unified Memory option with page migration through the CPU, then hardware GPUDirect P2P support was introduced in the Kepler microarchitecture with system agent bypass via PCIe switch, and the modern iteration with NVLink and NVSwitch provides a fully connected mesh topology with 900 GB/s bandwidth per GPU and atomic operations on remote memory without loading central processors.

Latest publications