VMEnter (CPU transition instruction to Guest)

VMEnter is the central hardware virtualization operation on Intel processors. It is not a software function but a processor instruction that transitions the CPU from hypervisor mode to guest virtual machine mode, handing over full control of computing resources without host operating system intervention.

This instruction is a key element of the VMX architecture in production server consolidation and cloud computing environments. It triggers every time a hypervisor, such as KVM or ESXi, decides to resume guest operating system execution after processing a system event. This happens after configuring the VMCS structure describing the guest state and is critical for delivering performance comparable to running on physical hardware.

VMCS (Virtual Machine control structure)KVM (Turns the Linux kernel into a hypervisor)

When using VMEnter, the main issue is the delay in processing pending kernel tasks. Since the scheduler sees entering the guest as a return to userspace, accumulated signals or work items may be ignored until the next exit, leading to real-time failures. Another difficulty is stack frame corruption, as the assembly code preparing for entry inevitably modifies the RBP register, disrupting kernel tracing mechanisms.

How VMENTER works

At the core of VMEnter is loading processor state from a pre-prepared data structure called the Virtual Machine Control Structure. The process can be roughly divided into several stages. First, the hypervisor performs checks to prevent entering the guest with inconsistent data. Then, the VMCS address is written to special processor registers, and the instruction itself is executed. The processor atomically switches context: the guest RIP instruction pointer is loaded from the VMCS, and control is passed to the guest OS without hypervisor involvement. When events requiring host intervention occur, such as hardware interrupts, guest execution is suspended, and the processor performs a reverse transition, VMExit, saving the exit reason in the same control structure. Importantly, VMEnter is not just a jump to an address but a full kernel mode switch performed as efficiently as possible to minimize virtualization overhead.

VMExit (Hypervisor interception of VM control)

VMEnter functionality

  1. VMCS Structure and Launch Validation. Before executing a VM entry, the processor checks the current Virtual Machine Control Structure for consistency and proper filling of mandatory fields. The checks cover the guest area, host area, and control fields. If any condition is not met, the instruction causes VM-Fail, returning an error code without transferring control to the guest system.
  2. VMLAUNCH Instruction for Initialization. To perform a VM entry with a clean VMCS, the VMLAUNCH instruction is used, called via the __vmx_vmlaunch intrinsic. It is used strictly when first starting a virtual machine or after resetting the VMCS state with the VMCLEAR instruction. An attempt to execute VMLAUNCH on an active structure that has already been in non-root mode is detected by the hardware as an error.
  3. VMLAUNCH (Launching a guest virtual machine)
  4. VMRESUME Instruction for Re-entry. To re-enter a guest system after a previous VM exit, the VMRESUME instruction is used. It only works with a VMCS in the Launched state. The VMM chooses between VMLAUNCH and VMRESUME based on a launch state flag, which is typically cached and tracked in software within the hypervisor code.
  5. VMRESUME (Resuming a suspended virtual machine)VMM (Hardware resource isolation and emulation)
  6. Low-Level Entry Point vmx_vmenter. In the Linux kernel, the vmx_vmenter function is a wrapper that branches based on the VMCS state. It checks the LAUNCHED flag and executes either vmresume or vmlaunch accordingly. Upon successful VM entry, the physical processor immediately switches context to the guest, and return only occurs on a subsequent VM exit to the point specified in the HOST_RIP field.
  7. Events During VM-Entry Transition. Immediately before executing the VMLAUNCH or VMRESUME assembly instruction, the hypervisor must program specific VMCS fields to control behavior at entry time. Such fields include VM_ENTRY_CONTROLS, which determines whether to load debug registers DR7/DR0, transition to IA-32e mode, or activate loading of model-specific registers (MSRs).
  8. VM_ENTRY_CONTROLS Field. This 32-bit field manages entry micro-details. Its bits are strictly validated against processor capabilities read from MSRs (the IA32_VMX_TRUE_ENTRY_CTLS or IA32_VMX_ENTRY_CTLS sets). The configuration algorithm requires respecting allow and forbid masks: bits set to 1 in the lower part of the MSR must be set, and bits set to 0 in the upper part must remain 0 in the final field value.
  9. Injecting Interrupts and Exceptions. The VM_ENTRY_INTR_INFO_FIELD provides a mechanism to inject external interrupts, non-maskable interrupts, or exceptions directly into the guest context at entry. The vector, delivery type, and valid flag are set via a bit mask. If the most significant bit is cleared to 0, the field is considered invalid, and the processor ignores its contents, performing a normal start of guest code.
  10. Loading Guest State. The hardware VM entry automatically loads guest state from the VMCS guest-state area into the corresponding processor registers. The key pointer, Guest RIP, determines the starting point of code execution. The processor restores segment registers, control registers CR0, CR3, CR4, and the MSRs specified in the MSR load area, providing an isolated execution environment.
  11. Saving Host State. Simultaneously with loading the guest, the hardware saves the current hypervisor state into the VMCS host-state area. This is critically important for subsequent return: the stack pointer (Host RSP), instruction pointer (Host RIP), and basic control registers are saved. Thus, the VMM does not lose its execution context and can correctly handle any exit reason.
  12. Preemption Timer Priority. To implement precise time slicing or force an immediate exit, the VMX preemption timer is used. Setting the timer field to 0 architecturally guarantees that a VM exit will occur before any guest instructions are executed after a successful VM entry. This avoids a race condition when delivering external interrupts via Self-IPI when KVM is running at L0 level.
  13. Interrupt Blocking Before VM Entry. Intel VMX hardware logic blocks external interrupt processing immediately before executing VMLAUNCH/VMRESUME and removes the block only after successfully entering the guest. This behavior ensures that a signal from the interrupt controller is not processed during the transitional moment between environments. Interrupt handling occurs within the guest context, causing an immediate VM exit.
  14. Causes of VM Entry Failure. If the VMCS contains invalid settings, the VM entry instruction completes with the carry flag (CF) or zero flag (ZF) set. In the case of VM-Fail Invalid (CF=1), the error is recorded in the VM-instruction error field. For VM-Fail Valid (ZF=1), details are written to the VM-Exit Information structure. The VMM must handle both scenarios and prevent the entire system from crashing.
  15. System Calls in Microkernels. In microkernel OSes such as seL4, the VMEnter concept is exposed at the system call level as seL4_SysVMEnter. Message registers (MR) pass control information about the entry. When an error or fault occurs inside the guest, the microkernel fills these registers with information about the cause and qualification of the exception, allowing a userspace VMM component to handle the syndrome.
  16. Return from VM Entry in Microkernel. In the seL4 API, the return from the VMEnter system call encodes the result in the msgInfo register. The constant SEL4_VMENTER_RESULT_FAULT signals an exit due to an exception, while SEL4_VMENTER_RESULT_NOTIF indicates a notification was received. This approach turns a complex processor transition into a deterministic API, hiding hardware virtualization details behind a software abstraction.
  17. Configuring Guest Address Space. Before VM entry, the VMM must correctly configure guest page tables and Extended Page Table (EPT) shadow structures. The VMCS field responsible for the EPT pointer links the guest physical address space to the machine’s real physical memory. Without a valid EPT structure, the entry attempt will fail with an invalid guest state error.
  18. EPT (Hardware second-level memory address translation)
  19. Controlling IA-32e Mode. Bit 9 in the VM_ENTRY_CONTROLS field controls activation of 64-bit compatibility mode for the guest. When the IA-32e mode guest flag is set, the processor expects that the guest EFER.LMA and EFER.LME registers, as well as corresponding entries in guest CR0 and CR4, are in a consistent state. Mismatch leads to VM entry failure.
  20. Simulating Nested Virtualization. When nested virtualization is enabled, the VM entry flow becomes more complex. The L0 hypervisor handles L1 guest VM entry into L2 not directly, but by emulating VMCS reads and writes for L1. Delays in event injection due to lack of IPI delivery guarantees during nested launches are resolved using the hardware preemption timer, guaranteeing exit without executing L2 instructions.
  21. Virtual Machine Migration Difficulties. During live migration or when the scheduler changes the physical core, the VMCS state must be synchronized. The VMCLEAR instruction is mandatory to transition the VMCS to the Clear state before migration, after which a new __vmx_vmlaunch will be required on the new core. VMRESUME is not applicable because VMCS logic is tied to a specific physical processor.
  22. Updating VMCS Before Re-entry. Each VM-Exit/VM-Entry cycle requires the VMM to analyze the exit reason (Exit Reason) and modify the VMCS if necessary. Without software processing, interrupt enable bits or shadow copies of registers may become outdated. The vmread and vmwrite functions allow modifying instruction fields and event vectors before the next vmresume call.
  23. Dual Nature of the VMEnter Function. At the architectural level, VMEnter is not a single instruction but a generic operation of transitioning to non-root mode, implemented via two different opcodes depending on context. A software wrapper function hides this difference, providing a unified interface. Such a design minimizes the risk of using the wrong instruction and improves hypervisor code readability.
  24. VM-Exit Handlers as Continuation of VMEnter. A key feature of the Linux implementation is that the vmx_vmenter code is inseparably linked to the vmx_vmexit entry point. The vmx_vmexit address is written directly into the HOST_RIP of the VMCS. Successful entry means an automatic transition to the guest, and a subsequent exit hardware-redirects execution flow back to vmx_vmexit, from where control returns to high-level KVM code.
  25. Scheduler Fault Tolerance. In case of software errors in the kernel during VM entry preparation, Linux uses special exception fixup sections in the vmenter.S code. If an unexpected exception occurs during vmlaunch or vmresume (for example, due to stack corruption), execution is redirected to a safe handler that logs the error and prevents a double processor fault.
  26. Usage in Windows Guest Environment. The __vmx_vmlaunch intrinsic requires x64 architecture compilation and is available via the intrin.h header file in Microsoft Visual Studio. Third-party Windows virtualization drivers use this intrinsic to create lightweight security hypervisors, running a guest OS in a depowered environment and intercepting its privileged instructions via VM exits.

Comparisons

  • VMEnter vs VMLAUNCH. VMLAUNCH is used for the first startup of a virtual machine when its VMCS is in the Clear state. VMEnter represents the general concept of entering non-root mode, while VMLAUNCH is a specific instruction that initializes this process and transitions the VMCS state to Launched. Repeatedly calling VMLAUNCH without prior clearing (VMCLEAR) will result in an error.
  • VMEnter vs VMRESUME. VMRESUME is used to return to a virtual machine after it has been successfully launched, i.e., when the VMCS status is already marked as Launched. VMEnter describes any transition to the guest system, while VMRESUME is a specific command that refuses to work with a clean VMCS. Both instructions perform the same basic checks but require different initial virtual machine states.
  • VMEnter vs VMExit. VMEnter initiates transferring control from the VMM to the guest operating system, while VMExit is a forced exit from the guest environment back to the VMM. On VMEnter, host state is saved in the VMCS and guest state is loaded; VMExit performs the reverse actions for the hypervisor to handle sensitive guest instructions or raised exceptions. These mechanisms form a closed loop of hardware virtualization operation.
  • VMEnter vs VMRUN. VMEnter belongs to Intel VT-x technology, while VMRUN is the equivalent for AMD-V (SVM). VMEnter is implemented via the VMLAUNCH/VMRESUME instructions, while VMRUN uses a single command that takes the physical address of the VMCB control structure. Despite having the identical goal of launching non-root mode, these instructions are architecturally incompatible and depend on the processor type.
  • Intel VT-x (Hardware Virtualization of the CPU)AMD-V (Hardware virtualization using the processor)VMCB (Virtual Machine state data structure)SVM (Full hardware isolation of virtual machines)
  • VMEnter vs SYSENTER. SYSENTER is used for fast switching from user mode (Ring 3) to kernel mode (Ring 0) within a guest OS. VMEnter switches the processor entirely between VMX Root (host) and Non-Root (guest) modes. Unlike SYSENTER, which activates a system call handler, VMEnter loads the full hardware context of an independent virtual machine from the VMCS area, providing isolation at the hardware level.

OS and driver support

Support is not implemented via drivers inside the guest OS but through a hypervisor that emulates the hardware: the guest OS is loaded into non-privileged (Ring 0 without real hardware control), and the hypervisor intercepts (VM-Exit) all privileged instructions and accesses to ports/MMIO, after which device drivers (virtio, emulated e1000, etc.) operate on top of paravirtualized or fully emulated buses and controllers provided by KVM.

Security

Isolation and protection are based on hardware extensions (VMX): before VMEnter, the hypervisor configures the VMCS, defining access maps for memory (EPT/NPT), interrupts, and IO ports, and to neutralize vulnerabilities like L1TF, a flush of the L1 data cache is performed immediately before entering the guest if the processor lacks hardware protection (X86_BUG_L1TF), guaranteeing that host or other virtual machine data cannot leak via side-channel attacks.

NPT (Second-level address translation for virtualization)

Logging

Event logging at the VMEnter stage is implemented via the kernel tracing mechanism (ftrace/tracepoints) and debugging hooks in the assembly entry code (vmx_vmenter): when VM-Fail occurs, the CF/ZF flags in RFLAGS are analyzed, and detailed failure reason information (VM-Exit reason, qualification) is extracted by reading VMCS fields and saved to the system log (dmesg) via pr_err or KVM statistics counters (debugfs), allowing identification of incorrect guest configuration or hardware faults without stopping the entire system.

Limitations

Hardware and software limitations stem from the VMX architecture: the processor cannot remain in guest mode with interrupts disabled for a long period without risking real-time timing violations (watchdog), nested virtualization depth is strictly limited (typically 1 level), and any delays before VMEnter, such as handling deferred work (task_work), must complete before the VMLAUNCH/VMRESUME command, because remaining in the VM-Entry state without exiting back blocks the host scheduler and critically important kernel timers.

History and development

The mechanism has evolved from simple wrappers for VMX instructions in Xen/KVM to deep integration with the Linux scheduler: whereas in early 2013 versions vmx_vmenter_helper was only used to set up state before the guest, modern kernels (starting with 2019 patches) added signal handling (TIF_SIGPENDING), cache flush requirements (L1TF mitigation), and handling of deferred RCU tasks directly in the vcpu_run loop to the VMEnter code, transforming the entry point into a full-fledged security and resource management gateway between host and guest.

Latest publications