VMRESUME (Resuming a suspended virtual machine)

VMRESUME is a hypervisor command that starts a virtual machine from a saved state file rather than performing a cold start of the operating system. The process restores memory contents, processor registers, and the state of emulated devices exactly to the point where the machine was previously suspended by the VMSUSPEND utility.

This technology is actively used by data center administrators for load balancing and physical host maintenance. Live migration of guest systems between servers without interrupting network connections becomes possible precisely through the cyclic call of saving and resuming. Developers use this mechanism to instantly freeze the environment at the stage of a difficult-to-reproduce error, and in large cloud platforms, memory snapshots allow reallocating resources between tenants during peak load periods.

The main difficulty involves loss of context for external devices passed through directly via PCI Passthrough. Graphics accelerators and physical-level network adapters may not support reinitialization without a full reset, causing drivers inside the guest system to hang. There is also the problem of time divergence: after a prolonged pause, clock synchronization services experience a huge discrepancy, which cascades and breaks Kerberos authentication mechanisms. The most critical risk remains the incompatibility of the saved state format when updating the hypervisor version, rendering old recovery files completely unreadable.

How VMRESUME works

The operating principle is based on capturing and serializing the entire deterministic state of the virtual platform. When the operator initiates saving, the hypervisor suspends the virtual central processing units and copies the guest RAM contents along with virtual chipset metadata into a single highly compressed binary file. When VMRESUME is called, the hypervisor performs reverse deserialization: it loads the memory image layer by layer into a pre-created virtual machine skeleton with an identical configuration, restores the saved values of model-specific registers MSRs and segment selectors, and the system clock emulation block compensates for the time gap by instantly injecting a corrected value. Then the hypervisor restores interrupted I/O processes and re-registers event handlers. A key feature is the impossibility of editing the saved state – the image checksum is strictly verified before startup to prevent corruption of operating system kernel structures due to injection of incorrect data into restored memory.

VMRESUME functionality

1. Virtual machine initialization after hibernation. VMResume is called by the hypervisor to restore the virtual machine state from a saved state file. The procedure starts when the operating system kernel detects a resume flag in the boot parameters passed by the Windows boot manager.
2. Reading and validating the saved context. The function reads the metadata block from hiberfil.sys, verifying checksums and magic signatures. The mechanism ensures the memory and processor register image is not corrupted. If inconsistencies are detected, VMResume initiates an emergency stop with an error code, preventing an unstable system from starting.
3. Decompressing the physical memory dump. Memory pages compressed with the Xpress algorithm are decompressed and copied into allocated ranges of physical RAM. The process includes reversing the compression algorithm used by the VmSave component. Restoration occurs taking into account caching attributes and access flags set before entering sleep mode.
4. Restoring address translation tables. The procedure loads saved Second Level Address Translation SLAT structures directly into the processor’s EPT/NPT control registers. This allows the hypervisor to immediately begin correct translation of guest physical addresses to machine addresses without going through a full cycle of building tables from scratch, bypassing page faults.
5. NPT (Second-level address translation for virtualization)EPT (Hardware second-level memory address translation)
6. Reinitializing the VP device. VMResume programs the virtual processor VP with values from the saved context. Segment registers, the interrupt descriptor table, control registers CR0, CR3, CR4, and MSR areas are loaded. The command pipeline state is precisely restored to ensure execution atomicity at the moment of suspension.
7. Restarting virtualized timers. The function reprograms the local APIC timer emulation and the Hyper-V reference time counter. The time gap occurring during the OS sleep period is compensated. A real-time delta formula is applied to prevent desynchronization of guest time synchronization services and the thread scheduler.
8. Reconstructing synthetic device state. VMBus interfaces restore connections with synthetic guest OS drivers. VMResume initiates protocol version negotiation and data channel reconnection procedures for network, storage, and video. I/O buffers that were being processed at the time of suspension are flushed or resent.
9. VMBus (Hypervisor-to-Guest communication channel)
10. Handling VPCI emulation state. For devices passed through via discrete descriptor, the PCI Express context is restored. VMResume reinitializes configuration spaces and remaps MSI/MSI-X interrupts, matching them to current APIC vectors. The operation is performed atomically to prevent loss of interrupts from direct access devices.
11. Restoring VP security context. Saved isolation tokens based on Virtual Secure Mode are loaded. VMResume verifies the authenticity of the protected kernel snapshot and restores access rights to VTL1 protected memory regions. Any discrepancy in signature or cryptographic hashes causes immediate termination of the recovery process.
12. Correcting the physical memory map after ACPI wakeup. After power is applied, the firmware may have altered the e820 memory map. VMResume analyzes intersections of restored pages with new firmware-reserved areas and performs relocation of conflicting guest physical blocks, updating SLAT tables to reflect these changes.
13. Checking for hot-swap capability. The function compares the PCI device topology at wakeup with a reference snapshot taken during saving. If removal of a VPCI device critical for boot is detected, VMResume generates a fatal error. When new devices are added, the hot-plug event is postponed until full OS recovery.
14. Synchronizing Non-Uniform Memory Access nodes. In NUMA topology systems, page affinity to physical processor sockets is restored. VMResume ensures that pages belonging to a specific proximity domain are restored to memory local to the corresponding VP. Violating locality would cause significant performance degradation.
15. Validating failover cluster state. In a guest cluster environment with shared storage, VMResume performs node reintegration. The function initiates SCSI reservation reset on shared volumes and activates the quorum rebuild timeout. This prevents split-brain state and protects cluster shared volume file system CSVFS integrity.
16. Flushing and restoring the TLB. All processor translation lookaside buffers are flushed. Selective TLB filling is performed with entries critical for the VMResume entry point code and recovery stack. INVVPID and INVEPT instructions are used to guarantee address mapping coherence in shadow mode.
17. Setting up the recovery watchdog timer. The function programs a hardware NMI timer in case the recovery process hangs. If VMResume does not transfer control to the kernel entry point within a strictly allotted interval, the timer initiates a non-maskable interrupt request, transitioning the system to a crash memory dump instead of infinite hanging.
18. Reconfiguring the power management subsystem. VMResume interacts with the hypervisor ACPI emulation, updating power button and sleep state status. The system is informed that a transition from S4 state has occurred. Wakeup registers are reset, PM1 status flags are cleared, allowing guest drivers to correctly handle subsequent sleep requests.
19. Handling page decompression errors. If a corrupted fragment is detected in hiberfil.sys, the Xpress-Huffman algorithm generates an exception. VMResume catches it via the hypervisor’s Structured Exception Handling mechanism, marking the damaged page as invalid and injecting an artificial machine check into the guest to isolate the owner process.
20. Injecting the IRP Power Sequence event. To resume bus driver operation, upon completing VMResume the hypervisor sends a Plug and Play IRP with the SystemPowerStateResume code. This initiates device enumeration without a physical bus reset. File system filter drivers receive a command to lift the I/O operation freeze.
21. Unlocking the guest OS scheduler. The final phase of VMResume releases the kernel object dispatcher lock. Exit from the hibernation context is performed by loading the stack pointer into the RSP register and executing the interrupt exit instruction. Control is atomically transferred to the saved instruction pointer of the KiRestoreFastResume entry point.
22. Releasing hypervisor context. After successful transfer of control to the guest OS, VMResume frees temporary allocations created for recovery buffers and unmounts scratch pages. The save state management structure is destroyed, returning resources to the parent partition and preventing hypervisor memory leaks.

Comparisons

• VMRESUME vs Resume-VM. The VMRESUME function resumes a virtual machine from a suspended state by loading a previously saved virtual machine file .vmss. In contrast, the Resume-VM operation in PowerShell starts a suspended VM without needing to specify a state file, as the hypervisor automatically tracks its current status, making the scenario more convenient for scripted automation.
• VMRESUME vs VMRESUMEHARD. The key difference lies in checkpoint handling. Standard VMRESUME uses basic standard checkpoints, restoring memory and device state. VMRESUMEHARD, conversely, initiates a hard resume, discarding saved memory state data and effectively performing a cold start of the guest OS, useful when the save file is corrupted.
• VMRESUME vs VMPOWERON. VMRESUME restores exactly the VM state captured at the moment of suspension Suspend, including RAM contents. The VMPOWERON function performs a normal start of a powered-off machine, going through the full POST cycle and operating system boot. Thus, VMRESUME provides instant continuation of application work, while VMPOWERON starts a clean session.
• VMRESUME vs VMMIGRATE. Although both operations aim to restore VM availability, their context differs. VMRESUME works locally within a single host and resumes operation after manual or automatic pause. VMMIGRATE is used during live migration between hosts, where memory state is synchronized and the VM resumes on the target node without visible service interruption.
• VMRESUME vs VMREVERT. The VMREVERT operation returns a virtual machine to a previous snapshot point, rolling back disk, memory, and settings to a historical snapshot, which is destructive to current data. VMRESUME does not alter the VM execution timeline; it simply unpauses it, preserving all information since the stop and providing linear continuation of work without losing changes.

OS and driver support

VMRESUME implements guest operating system support through the paravirtualized driver mechanism VMBus, which is built into the Linux kernel starting from version 4.x and available for Windows through Integration Services, while virtual machine wakeup after hibernation or state saving is achieved by precisely restoring the processor context MSR registers, CR modes and reinitializing emulated I/O devices via the /dev/kvm ioctl interface, guaranteeing atomic APIC timer restoration and reconnection of virtual network adapters without breaking TCP sessions.

Security

VMRESUME context isolation is ensured via hardware virtualization mechanisms Intel VT-x VMCS and AMD-V VMCB, where upon VM resume, checksums of guest memory pages SEV-SNP attestation for encrypted machines are validated, snapshot integrity Launch Digest is verified, and hypervisor-level RBAC policies are applied, filtering VMRESUME calls via KVM_CAP_MANUAL_DIRTY_LOG_PROTECT to prevent VENOM-class attacks through corrupted virtual device buffers.

Logging

The VMRESUME logging subsystem is based on the ftrace channel in the Linux kernel, where each call to the kvm_arch_vcpu_ioctl_run function marks the entry point in the tracer, and detail is provided via KVM_EXIT_REASON codes KVM_EXIT_SHUTDOWN and KVM_EXIT_INTERNAL_ERROR are specially handled with output to the debugfs interface /sys/kernel/debug/kvm/*, supplemented by a userspace daemon that aggregates vmexit events by cause EPT Violation, CR Access and stores them in structured form in journald with binding to VCPU-ID and TSC timestamp.

Limitations

Technical limitations of VMRESUME are determined by the impossibility of restoring direct PCI device passthrough VFIO without a full device reset on the bus, loss of HARDENED_USERCOPY performance counters during migration between processors with different microarchitectures in a cluster, as well as the critical requirement for identical SMM System Management Mode configuration and platform microcode, which in practice forces the use of CPU Pinning taskset and prohibits live migration between hosts with different XSAVES/MPX flags in cpuid, restricting the application scenario to static infrastructure without hot-remove of resources.

History and evolution

The evolution of VMRESUME started with patches from Avi Kivity for KVM in 2006 as a simple VMLAUNCH/VMRESUME instruction in ring 0 opaque, then transformed through the introduction of the KVM Forum save/restore API specification KVM_GET/SET_REGS, KVM_GET/SET_SREGS, gained support for nested virtualization VMCS Shadowing, Intel eVMCS in kernel 4.20 with optimization via hardware vmcs12 cache, and in modern builds uses SEV-ES Encrypted State for full encryption of the VCPU register file upon resume, reflected in the vmx_vmresume and svm_vmrun macros integrated into KVM_VCPU_INIT with automatic capability detection via CPUID.0x8000000A.