SR-IOV (Hardware-level input-output device virtualization)

SR-IOV (Single Root I/O Virtualization) allows a single physical PCIe device such as a network card to present itself to multiple virtual machines as a set of independent fast devices, bypassing the hypervisor software layer.

This technology is used in data centers for high-performance networks (10/25/100 Gbit/s) and NVMe storage. It is in demand in NFV environments (network function virtualization), cloud platforms (OpenStack, VMware), and for latency-critical applications such as real-time databases or trading.

Typical issues

The main problem is limited scalability due to a fixed number of virtual functions per port. Difficulties arise with live migration of virtual machines because a virtual function is tightly bound to a physical device. Guest driver support is also required, and incorrect behavior by a virtual machine can cause the entire physical adapter to fail.

How SR-IOV works

An SR-IOV capable device creates two types of resources inside itself: one Physical Function (PF) and many Virtual Functions (VFs). The PF is managed by the host system driver and is responsible for global card settings, firmware, and virtualization. Each VF is a lightweight, isolated hardware context with its own queues, registers, and DMA mechanisms. The hypervisor assigns a VF directly to a guest virtual machine via the IOMMU (address translation and memory protection), after which traffic flows from the VM to the device without hypervisor involvement. The hardware switch inside the adapter then routes packets between VFs and the physical port itself, providing line rate and zero software processing latency.

IOMMU (Isolation of direct memory access addresses)VF (Hardware I/O virtualization mechanism)PF (Hardware virtualization of Input-Output devices)

SR-IOV functionality

  1. Physical function. The physical function (PF) is the main component of an SR-IOV device. The PF is managed by the hypervisor or host driver. It handles global configuration, virtual function setup, power management, and device-level error handling.
  2. Virtual function. The virtual function (VF) is a lightweight representation of a PCIe device. A VF is assigned directly to a virtual machine without hypervisor involvement during data transfer. Each VF has a unique identifier (Bus:Device.Function) and a full PCIe transaction stack.
  3. MMIO and BAR registers. Each VF has its own Memory-Mapped I/O region mapped into the root system address space. VF BAR registers contain addresses for descriptor rings and control registers, which the VF accesses directly without hypervisor interception.
  4. Assignment to a virtual machine. A VF can be passed to a guest OS via the PCI passthrough mechanism (e.g. VFIO in Linux). The guest driver interacts with the VF as an ordinary PCIe device, performing MMIO mapping and interrupt handling without hypervisor intermediaries.
  5. VFIO (Direct device Input-Output virtualization)
  6. Hypervisor support. Modern hypervisors (KVM, Xen, ESXi, Hyper-V) implement SR-IOV via virtual switches and PF management modules. The hypervisor reserves VFs, creates a virtual bridge, and proxies access control settings but does not participate in data transfer.
  7. Interrupts and MSI-X. VFs support extended Message Signaled Interrupts (MSI-X). Interrupts from a VF are delivered directly to the virtual machine via the IOMMU remapping table, providing low latency and reducing CPU load.
  8. Device reset handling. When a virtual machine reboots or a VF encounters an error, the functional state needs to be reset. Three types are supported: Function Level Reset (FLR) via a PCIe control register, reset of the entire PF (affects all VFs), and secondary bus reset.
  9. IOMMU and address translation. To isolate DMA operations of a VF, an IOMMU (Intel VT-d, AMD-Vi) is required. The IOMMU translates guest physical addresses to machine addresses, preventing incorrect DMA access from a VF into the memory of other VMs or the hypervisor.
  10. VF count limitation. The maximum number of VFs is set by the device manufacturer and limited by internal resources: number of queues, DMA contexts, interrupts, and internal memory. The typical range is from 4 to 256 per port.
  11. Classic device examples. SR-IOV is primarily used in network adapters (Intel XL710, Mellanox ConnectX, Broadcom NetXtreme) and storage controllers (NVMe SSDs with VF support). GPU implementations of SR-IOV are more complex due to shared memory contexts.
  12. Virtual functions in networks. For Ethernet adapters, each VF receives a separate MAC address and can use VLAN, QoS, and offload functions (LRO, TSO, RSS). The hypervisor often creates a VF representor for flow management in the host switch.
  13. SR-IOV and hardware queues. Modern network cards have multiple receive/transmit ring buffer pairs. When a VF is created, each queue is hard‑assigned to a specific VF, eliminating contention and the need for arbitration by the PF.
  14. Risk of disconnected management. Because the hypervisor does not intercept the data path, it loses the ability to monitor traffic between VFs. To control performance and security, methods such as VF‑Tagging, port mirroring on the physical switch, or in‑VM agents are used.
  15. Root complex requirements. For correct SR-IOV operation, chipset and processor support is required: ACS (Access Control Services) extension for function isolation, correct TLP (Transaction Layer Packet) routing, and a sufficient number of PCIe root ports.
  16. Linux configuration. Enabling SR-IOV in the Linux kernel is done using the ip link set dev eth0 vf 0 mac aa:bb:cc:dd:ee:ff command or with vfio‑pci modules. The number of VFs is set via sysfs: echo 8 > /sys/class/net/eth0/device/sriov_numvfs.
  17. VFIO and userspace access. The VFIO module provides secure access to VFs from unprivileged userspace via IOMMU groups. The VFIO‑pci driver binds a VF, sets up interrupts and memory, and prevents direct access to the PF.
  18. Differences from device emulation. Unlike e1000 or virtio‑net, a VF does not require data copying or context switching. Packet transfer latency through SR‑IOV drops from several microseconds (emulation) to sub‑microsecond values (at the DMA level).
  19. Virtual machine migration problem. Because a VF holds hardware state on the physical device, live migration of a VM with an assigned VF is extremely difficult. Solutions involve detaching the VF, falling back to emulation during migration, and re‑attaching on the target host.
  20. Conflict with SR‑IOV on a single port. If two applications from different VMs try to simultaneously configure critical registers of the same physical function (e.g. port speed control), an external coordinator may be needed: a PF driver that blocks conflicting requests from VFs at the firmware level.

Comparisons

  • SR-IOV vs Virtio. SR-IOV provides dedicated virtual functions (VFs) for direct hardware access bypassing the hypervisor, delivering near‑native performance. Virtio, in contrast, uses paravirtualization with a shared backend and requires hypervisor involvement in I/O, trading speed for universal compatibility and ease of live migration.
  • SR-IOV vs PCI Passthrough. PCI Passthrough hands over an entire physical device to a single VM, excluding resource sharing. SR-IOV splits one device into many lightweight VFs, allowing a port to be shared among dozens of VMs. At the same time, SR-IOV reduces domain management overhead and preserves isolation, but requires hardware support from the NIC/GPU.
  • SR-IOV vs OVS-DPDK. OVS‑DPDK accelerates virtual switches by bypassing the kernel but still keeps the packet path through the hypervisor. SR‑IOV eliminates the hypervisor from the critical path entirely: packets go straight into the VM. However, OVS‑DPDK offers flexible software processing (GRE, VXLAN, tagging), while SR‑IOV relies on static hardware capabilities.
  • SR-IOV vs Virtio + vhost‑user. Virtio with vhost‑user uses shared memory and dedicated threads for queue handling, avoiding frequent context switches, but the hypervisor still manages the queues. SR‑IOV goes further by removing any intermediate host processing. At the same time, vhost‑user supports live migration without driver reset, whereas SR‑IOV requires special device support and often loses state.
  • SR-IOV vs Virtio‑net with hardware offload. With a SmartNIC and Virtio offload support, a paravirtual interface can be embedded directly into hardware, combining the benefits of standard Virtio drivers with acceleration via an SR‑IOV‑like mechanism. However, classic SR‑IOV is simpler to configure and does not require guest driver modifications, while offload solutions provide better migration compatibility but are more complex to implement at the switch level.

OS and driver support

SR-IOV requires support at the hypervisor level (KVM, Xen, ESXi) and in the guest OS: the physical function (PF) is managed by the host system driver, which creates virtual functions (VFs); inside the guest OS, a standard device driver (e.g. ixgbevf, iavf for Intel Ethernet) works directly with the VF via PCIe, bypassing the hypervisor. Modern Linux kernels (since 2.6.37) and Windows Server (since 2012 R2) have built‑in support.

Security

Hardware separation of VFs via PCIe Access Control Services (ACS) and isolation of DMA transactions via IOMMU (VT‑d or AMD‑Vi) guarantee that a virtual machine accesses only its own buffers and registers. A malicious VF cannot read or modify data of another VF or the PF, and the hypervisor further restricts configuration spaces through SR‑IOV Extended Capability mechanisms.

Logging

Logging in SR-IOV is implemented at three levels: the PF driver logs VF creation/destruction and initialization errors to the system log (dmesg, Event Viewer); additionally, hardware failures (e.g. transaction completion with an error) are recorded in PCIe AER (Advanced Error Reporting) extended status registers; in the guest OS, the VF driver writes events to its own log, and the hypervisor can aggregate logs from all VFs via virtual diagnostic mechanisms.

Limitations

Key limitations: the maximum number of VFs is fixed by the manufacturer (typically 64–256 per port); live migration of a VM with a directly assigned VF is impossible without detaching the device and losing state; moreover, not all devices support SR-IOV (hardware implementation required), VFs do not support arbitrary load balancing or advanced power management features, and reallocating resources requires a PF reset, interrupting all associated VMs.

History and evolution

In 2007, PCI‑SIG ratified the SR-IOV 1.0 specification as part of PCIe 2.0, and Intel and AMD added support in their chipsets. Subsequent development focused on improving scalability (increasing the number of VFs in PCIe 4.0/5.0), introducing Scalable IOV (S-IOV) for more flexible resource allocation, integration with NVMe and GPUs (NVIDIA vGPU with SR-IOV), and combining SR-IOV with VFIO and DPDK to achieve near‑native performance.

vGPU (Splitting a GPU into virtual devices)