Xen Hypercall (Cross-guest call to Xen hypervisor)

Xen Hypercall is a system call from an unprivileged virtual machine (domain) to the Xen hypervisor to perform privileged operations such as memory management or sending interrupts.

Hypercalls are used in paravirtualized guests (PV mode) and in PVH mode (without BIOS emulation), where domain drivers interact directly with Xen. They are also used by the control domain (Dom0) to create and configure other guests, migrate virtual machines, and work with virtual devices via split drivers.

Dom0 (Xen Virtual Machine management)PV (Virtual machine I/O acceleration)

Typical issues include erroneous passing of pointers to memory that has not been validated (leading to hypervisor crash or DoS), as well as vulnerabilities such as hypercalls with incorrect data length (e.g., CVE-2017-7228). Additionally, wasteful hypercalls from a malicious domain can monopolize the physical CPU due to insufficient quota control.

How Xen Hypercall works

The working principle is based on a context switching mechanism with protection. When a guest attempts to execute an instruction (e.g., syscall or int 0x82 on x86, or the ARM hvc special instruction), the processor automatically raises the privilege level from unprivileged guest code (Ring 3 for PV, Ring 0 for PVH with restrictions) to the hypervisor level (Ring -1 or EL2). Xen takes control, reads the hypercall number from a register (e.g., EAX on x86) and the parameter array from other registers or from guest memory. The hypervisor checks whether the requested number falls within a valid range, whether all passed pointers point to memory of the current domain and do not overlap with Xen metadata. Xen then temporarily maps the guest’s pages into its own address space (or uses a mapping mechanism) and performs the required operation — updating page tables (e.g., do_mmu_update), sending an event (EVTCHNOP_send), or changing virtual CPU state (VCPUOP_*). Before returning the result, Xen flushes temporary mappings, updates the guest state (if EFLAGS or the return register needs to be changed), and executes a return instruction, switching context back to the guest domain. Thus, the hypercall serves as a strictly controlled and only legal channel for requesting services from the underlying hypervisor.

Xen Hypercall functionality

  1. Switching mechanism. Execution of a hypercall is initiated by a special instruction (e.g., syscall or int on x86) that switches context from protection ring 3 (guest) to ring 0 (hypervisor). Xen checks the operation code and parameters.
  2. Call identification. Each hypercall is identified by a 32-bit number passed through a register (usually EAX/RAX). Up to six arguments are passed through other general-purpose registers (RDI, RSI, RDX, RCX, R8, R9).
  3. Handling in the hypervisor. Upon entering Xen, dispatching occurs via the hypercall_table[]. Each handler checks the validity of passed pointers using the guest_handle mechanism to avoid faults when accessing guest memory.
  4. Classification by area. Hypercalls are divided into categories: domain management (DOMCTL), memory management (MMU, MEM), event handling (EVTCHN), I/O operations (PHYSDEV, HVM), as well as debugging and auxiliary functions.
  5. HVM (Hardware isolation and virtualization acceleration)
  6. Hypercalls for HVM guests. For fully virtualized guests (HVM), Xen uses emulation via hvm_hypercall. The call is made through the hypercall_page, which the hypervisor maps into the guest address space.
  7. The hypercall page. PV guests receive from Xen a page with call templates (xen_hypercall_table) containing instruction sequences for fast switching. This page is available at a fixed virtual address.
  8. Buffer passing. For large amounts of data, hypercalls use guest_handle descriptors that point to a buffer in guest memory. Xen copies data into its internal space using copy_from_guest and copy_to_guest macros.
  9. Security checks. Before execution, Xen checks that all pointers belong to the current domain and do not overlap with the critical area of the hypervisor. Violation results in a #GP and termination of the hypercall with an error code.
  10. DOMCTL hypercalls. DOMCTL operations manage domain lifecycle: creation (XEN_DOMCTL_createdomain), destruction (XEN_DOMCTL_destroydomain), parameter configuration (memory, CPUs, limits). They require administrator privileges.
  11. MEMOP hypercalls. The HYPERVISOR_memory_op group allows modifying page tables and memory mappings. Operations include adding/removing pages (XENMEM_add_to_physmap, XENMEM_remove_from_physmap), changing memory type, and setting protection.
  12. Event channels. HYPERVISOR_event_channel_op manages asynchronous notifications between domains and the hypervisor. Operations include EVTCHNOP_alloc_unbound, EVTCHNOP_send, EVTCHNOP_bind_interdomain, EVTCHNOP_close.
  13. Hardware access hypercalls. HYPERVISOR_physdev_op is used for direct management of physical devices (e.g., IRQ, PCI configuration space). HVM guests with PCI pass-through use this hypercall for interrupt mapping.
  14. Debugging hypercalls. The HYPERVISOR_kexec_op and HYPERVISOR_debug_keys groups allow dumping hypervisor state, collecting traces, and forced crash with buffer preservation. They are used by Xen developers and the xen-dump toolset.
  15. Timing management hypercalls. HYPERVISOR_sched_op requests the hypervisor to switch context, yield, or change scheduling parameters. Key codes: SCHEDOP_yield, SCHEDOP_block, SCHEDOP_shutdown.
  16. Asynchronous completion. Some hypercalls (e.g., memory operations) may not complete instantly. Xen returns an -EAGAIN code, and the guest must retry the call after the next event occurs.
  17. Migration hypercalls. The XENMEM_set_memory_map and XENMEM_exchange hypercalls allow saving and restoring a domain’s memory image during live migration. They work in conjunction with dom0 tools.
  18. Return code conventions. Hypercalls return 0 on success, a negative standard kernel error code (-ENOMEM, -EINVAL, -EFAULT) on failure. HVM guests use RAX register modification with adjustment in the presence of I/O emulation.
  19. Recursion protection. The hypervisor forbids executing hypercalls from the context of hypervisor interrupts or while handling another hypercall for the same domain. This prevents recursive locks and stack overflow.
  20. Performance optimization. For frequent operations (getting time, retrieving system parameters), there are special hypercalls with minimal checking. For example, HYPERVISOR_get_debugreg does not switch address space and executes in tens of cycles.

Comparisons

  • Xen Hypercall vs Linux syscall. A hypercall switches the guest OS directly into Xen hypervisor management mode via vmcall/syscall, bypassing the host OS kernel, whereas a syscall is translated through VDSO and an exception handler into the Linux kernel, adding emulation overhead in virtualization.
  • Xen Hypercall vs KVM ioctl. A hypercall is executed synchronously from the guest without context switching to the host process level, while an ioctl in KVM requires a transition through the /dev/kvm file descriptor, a system call, and a return to QEMU, increasing latency for frequent calls.
  • Xen Hypercall vs VMware hypervisor (VMCALL). Xen uses a direct int $0x82 or syscall mechanism with a fixed port, whereas VMware VMCALL emulates a CPU instruction with dynamic trapping, requires vmcb analysis, and has higher latency for privileged operations.
  • Xen Hypercall vs Hyper-V hypercall. Xen hypercalls provide up to 32 parameters via registers and have a flat code space; Hyper-V uses an in-memory hypercall input/output structure with PA and a mandatory call code field, which complicates serialization but improves scalability.
  • Xen Hypercall vs SCOM call (PowerVM). Hypercalls are oriented toward x86/ARM with strong typing and domain checking, while SCOM uses asynchronous queues and real memory mapping, allowing batch processing of up to 16 calls per transition — winning in throughput but losing in determinism.

OS and driver support

Xen hypercall is implemented as an interface between guest OSes and the hypervisor via software interrupts (int 0x82 on x86 or hypercall page on ARM), where paravirtualized drivers (e.g., PV drivers for block devices and networking in Linux, FreeBSD) use hypercalls to interact directly with Xen instead of hardware emulation, while fully virtualized OSes (HVM) require drivers with a hypercall interface, such as XenWindowsDrivers, to provide high-performance I/O and memory management via do_mmu_update, update_va_mapping, and grant_table_op hypercalls.

Security

Xen controls access to hypercalls through bitmasks in the struct domain, checking hypercall arguments (e.g., set_gdt, set_trap_table) for address validity and access rights before execution, and mechanisms such as XSM (Xen Security Modules) and Flask allow forbidding or permitting individual hypercalls for unprivileged domains (DomU) and restricting Dom0, while hypercalls that manage memory (memory_op, xenmem_add_to_physmap) require explicit privilege level confirmation and verify that the guest does not modify critical hypervisor structures.

DomU (Automatic network port binding)

Logging

Xen hypercall logs its calls via built-in tracing mechanisms (xen/trace.h, TRACE_HYPERCALL in debug builds), where each hypercall, its operation code, and arguments can be written to a ring trace buffer and then passed via the xen_version or kexec hypercall to Dom0; tools such as xentrace and xenalyze allow administrators to monitor hypercall frequency and sequence for driver debugging and performance analysis without stopping the server.

Limitations

The main limitation is that not all guest OSes support paravirtualization due to the need for special hypercalls and kernel modification (e.g., Windows without PV drivers uses only a minimal set of HVMOP_* hypercalls), as well as hard limits on argument size (maximum of 6 scalar registers on x86_64; the rest must be passed via buffers with bounds checking). Moreover, hypercalls cannot be invoked from ring 3 (user mode) in the guest OS due to lack of necessary privileges, restricting their use from unprivileged applications without kernel mediation.

History and development

The Xen hypercall appeared in 2003 as a set of 40–50 basic calls (e.g., set_timer_op, event_channel_op, sched_op) for the first version of Xen on x86-32; then with each release (Xen 3.x added hypercalls for NUMA and grant tables, Xen 4.x for ARM via arm_hypercall_table and PVH mode, and Xen 4.13+ introduced hypercalls for protection against L1TF and MDS vulnerabilities, such as xen_hypercall_msr). The modern interface includes more than 150 operations supporting x86_64, ARMv7, and ARMv8, with a trend toward unifying calls via hypercall_table and reducing the use of obsolete hypercalls like physdev_op in favor of new ones (dm_op, hypfs).