Hyper-V virtualisation
Placing workloads between virtual machines and Kubernetes, preparing node machines, VHDX disk growth, and how Hyper-V checkpoints work and where they stop being a safeguard.
Hyper-V is a hypervisor that runs classic Windows and Linux servers as well as the node machines of a Kubernetes cluster. This document describes how workloads are split between virtual machines and containers, the settings of Kubernetes nodes on Hyper-V, the mechanism behind VHDX disk growth, and how checkpoints work and where their limits as a safeguard lie.
Workload placement: virtual machine or container
“Migrating from virtual machines to containers” is a false choice: the cluster itself runs on virtual machines, and some workloads should never go into containers at all. A domain controller cannot be containerised in any supported way, a decade-old application may have a licence tied to a hardware identifier, and an image registry running in the cluster is unavailable exactly when the cluster has to be rebuilt. Placement is decided by three questions:
- How often does it change? A release once a quarter or less often: a virtual machine. Several releases a week: a container, because declarative deployment saves time on each one.
- Who maintains it? If the team knows Windows Server but not Kubernetes, moving the workload will lower its availability.
- What happens when the platform goes down? A workload needed to repair the platform cannot run on it.
| Workload | Where | Why |
|---|---|---|
| Domain controller, file server | VM | full Windows, fixed machine identity, integration with the directory and NTFS permissions, no supported container variant |
| Legacy application, software licensed per host or core | VM | configuration in the registry, licence tied to a hardware identifier or the number of cores, a release once a quarter |
| Image registry, Git server, CI, backup server | VM, at least a copy outside the cluster; backups on a separate host | needed to rebuild the cluster: without a registry images cannot be pulled, without the repository there are no manifests; backups must survive the failure of everything they protect |
| API, stateless web front end, queue workers, new internal service | container | horizontal scaling, frequent releases, no migration cost |
| Database | VM, unless the team can rebuild it in the cluster | operators have matured, but need knowledge of data storage and rehearsed failures |
The database row is conditional: an operator in the cluster gives automatic failover, but a database nobody can recreate by hand in a container is safer on a machine with a backup every administrator can restore.
A Kubernetes node as a virtual machine
Several settings that suit application servers harm nodes:
| Setting | Value for a node | What happens with a wrong setting |
|---|---|---|
| Memory | static, e.g. 8 GB | Kubernetes places pods according to the memory reported by the node; dynamic memory quietly takes it away, resulting in the OOM killer and pod evictions |
| Checkpoints | disabled, including automatic ones | restoring one rolls one member’s etcd data back in time and it no longer matches the rest |
| Virtual processors | fixed number, no contention with other machines | a control plane node competing for CPU slows etcd and the API |
| Generation | 2; for Linux the Secure Boot template “Microsoft UEFI Certificate Authority” | the “Microsoft Windows” template is meant for Windows operating systems, while Linux distributions use “Microsoft UEFI Certificate Authority” |
| Time | a single source for all nodes | clock skew causes TLS errors and etcd clock drift warnings |
| Placement | at most one control plane node per host | the failure of a host with two etcd members takes away quorum |
| Disk | fast storage, no neighbours with write-heavy workloads | etcd waits for a disk write on every change; a slow disk means a slow API |
Switching memory to static (Set-VMMemory -DynamicMemoryEnabled $false -StartupBytes 8GB) requires the machine to be off. The node is first drained of pods (kubectl drain) and brought back afterwards (kubectl uncordon), and control plane nodes are changed one at a time so etcd does not lose quorum. Checkpoints are disabled with Set-VM -CheckpointType Disabled -AutomaticCheckpointsEnabled $false. A checkpoint is not a node backup; cluster state is protected by an etcd backup, the manifest repository and volume backups.
The Time Synchronization integration service synchronises the guest clock with the host’s clock, and Linux nodes may also run a chrony daemon with its own NTP servers. All nodes use one variant: synchronisation from the host, or chrony with the same NTP servers as the domain controllers and the integration service disabled (Disable-VMIntegrationService).
Spreading nodes across hosts
Three control plane nodes with etcd tolerate the loss of one member, so a safe layout is three hosts with one control plane node on each; on two hosts the failure of the one with two nodes takes away quorum. In a Hyper-V failover cluster the rule is kept by the AntiAffinityClassNames property of cluster groups, which places machines with the same class on different nodes where possible. The rule is soft: with no free host the cluster will place two machines together, and a change takes effect only after the groups are restarted. The hard variant is the cluster property ClusterEnforcedAntiAffinity. Placement has to be checked, not only set.
$class = New-Object System.Collections.Specialized.StringCollection
$class.Add('k8s-control-plane') | Out-Null
foreach ($vm in 'k8s-node-01', 'k8s-node-02', 'k8s-node-03') {
(Get-ClusterGroup -Name $vm).AntiAffinityClassNames = $class
}
Network and identity
The nodes’ network is a separate segment with static addresses. Clients reach the cluster API through a virtual address, not through a single node. Traffic from the server network to services in the cluster goes through defined entry points (the ingress controller, LoadBalancer service addresses), never through pod addresses. Authentication to the cluster uses the same identity directory as the rest of the environment, because separate Kubernetes accounts are a separate thing to audit.
VHDX disks: sizes and the causes of growth
A fixed disk takes up its full capacity on the host from the start and does not grow. A dynamic disk starts as a small file and allocates space in blocks when the guest first writes to a given area. Three figures matter: the capacity seen by the guest, the size of the file on the host, and the data in use in the guest (df -h on Linux, Get-Volume on Windows). The first two are shown by Get-VHD:
| Field | Meaning |
|---|---|
Size | logical capacity seen by the guest |
FileSize | size of the file on the host volume; on a dynamic disk it never shrinks by itself |
MinimumSize | smallest capacity Resize-VHD can shrink the disk to (follows from the partition layout) |
BlockSize | allocation unit, 32 MB by default for a dynamic disk |
A healthy dynamic disk has a FileSize only slightly larger than usage in the guest. A file close to Size while the guest is one-third full means freed space is not returning to the host.
Hyper-V sees only writes to sectors, not the guest’s file system. The first write to an area with no allocated block enlarges the file by a whole block, and deleting a file in the guest changes only metadata: the host is not told, so the block stays. If the guest always wrote to the same sectors, the file would stop at peak usage. File systems, however, place new data in different spots, and typical workloads constantly create new files and delete old ones: container image layers, rotated logs, the package manager cache and CI working directories, downloaded updates. After a few months the guest has written something to most areas of the disk and the file approaches full capacity at constant usage. Checkpoints are a separate growth mechanism, described below.
A 1 MB block for Linux guests
With a 32 MB block, a 4 KB write to a new area enlarges the file by 32 MB. This works in Windows guests, but ext4 and XFS spread metadata and data across the whole disk (block groups, allocation groups), so the file grows quickly despite little data. Microsoft recommends dynamic VHDX disks with a 1 MB block for Linux guests, ext4 instead of ext3 and mkfs.ext4 -G 4096.
A new disk is created with New-VHD -Path <new.vhdx> -SizeBytes 200GB -Dynamic -BlockSizeBytes 1MB, but the block size of an existing disk cannot be changed in place. The machine is switched off (no checkpoints, free space for the new file), the disk is converted with Convert-VHD -Path <old.vhdx> -DestinationPath <new.vhdx> -VHDType Dynamic -BlockSizeBytes 1MB and attached in place of the old one (Set-VMHardDiskDrive, same SCSI controller and position). The old file is deleted after confirming the machine boots from the new disk and has all its data.
TRIM and UNMAP: the guest reports free blocks
TRIM (ATA) and UNMAP (SCSI) tell the disk that a range of sectors no longer holds data; Hyper-V marks the corresponding VHDX blocks as unused. The conditions: a disk in VHDX format (VHD does not support UNMAP) on a controller that passes UNMAP through. A virtual SCSI controller is recommended (generation 2 machines have only SCSI, in generation 1 the system disk sits on IDE), and according to the documentation UNMAP also reaches the host through “enlightened IDE” and Virtual Fibre Channel.
On a Linux guest TRIM is sent by fstrim. In many distributions fstrim.timer runs weekly by default, in others it has to be enabled (sudo systemctl enable --now fstrim.timer); a one-off run is sudo fstrim -av. A Windows guest with NTFS sends delete notifications by default (on ReFS v2 TRIM is off by default), and the disk optimisation task repeats TRIM weekly. The setting is checked with fsutil behavior query DisableDeleteNotify (0 means enabled), and a manual run is Optimize-Volume -DriveLetter C -ReTrim. TRIM stops the growth, because freed blocks return to use, but it does not shrink the file.
Compaction and preventing growth
Optimize-VHD removes unused blocks from a dynamic or differencing disk (it does not apply to fixed disks). The disk cannot be in use by a running machine, and in Quick, Full and Retrim modes it has to be attached to the host read-only; Full also looks for blocks filled with zeros. The command shrinks FileSize, not Size: capacity is changed by Resize-VHD, and shrinking it requires shrinking the partition in the guest first, otherwise data may be lost.
Effectiveness depends on the guest file system. The host can read NTFS but not ext4 or XFS, so with a Linux guest the blocks reclaimed are those marked earlier by TRIM and zeroed ones; sudo fstrim -av is run just before switching the machine off.
# Linux guest: just before Stop-VM run sudo fstrim -av in the guest (not part of the script)
$vm = 'srv-app-01'
$vhdx = 'D:\VM\srv-app-01\srv-app-01.vhdx'
Stop-VM -Name $vm # shut down the guest operating system
$before = (Get-VHD -Path $vhdx).FileSize
Mount-VHD -Path $vhdx -ReadOnly # Full mode requires a read-only attachment
Optimize-VHD -Path $vhdx -Mode Full # removes unused and zeroed blocks
Dismount-VHD -Path $vhdx
Start-VM -Name $vm
'{0:N1} GB -> {1:N1} GB' -f ($before / 1GB), ((Get-VHD -Path $vhdx).FileSize / 1GB)
Compaction reclaims space once. To keep the file from returning to its previous size, the guest has to send TRIM regularly and not accumulate, without limits, data that will be deleted anyway: the system journal, application logs and container images get limits and clean-up (SystemMaxUse= in journald, log rotation, image clean-up thresholds in the kubelet). Limits and TRIM work only together: without TRIM, new files land in new areas and the VHDX still reaches full capacity, while TRIM without limits will not stop the guest from filling its own disk.
The minimum for monitoring is an alert on free space on the volume holding the machines, e.g. below 15%. An earlier signal comes from a job sending the possible growth of disks (Size minus FileSize for non-fixed ones) to monitoring. A file whose FileSize grows every week while usage in the guest stays flat points to a machine without working TRIM.
Checkpoints: the AVHDX chain
A checkpoint in Hyper-V is a tool for quickly rolling back a change, not a backup: it sits on the same storage as the machine, depends on the machine’s parent disk and costs space and performance with every hour. When a checkpoint is created, the machine’s VHDX disk stops being written to. Hyper-V creates a differencing disk with the .avhdx extension next to it, and from then on all writes go there. The next checkpoint creates another AVHDX file whose parent is the previous one. The machine runs at the end of the chain, and reading a block that has not changed reaches through the levels all the way to the base VHDX.
This design has three consequences:
- Space. Every AVHDX file grows with each block written and can grow to the size of the whole disk. Space is reclaimed only by deleting the checkpoint and merging the chain.
- Performance. Every additional level of the chain means extra work on reads; databases and file servers feel it first.
- Dependency. An AVHDX file without its parent is useless. Damage to the base VHDX, or to the volume it sits on, destroys the machine and all its checkpoints at once, which is what separates a checkpoint from a copy on a different system and medium.
Checkpoint types
| Type | What it captures | Application consistency | After applying |
|---|---|---|---|
| Standard | disks plus the machine’s memory and device state | none, the application is unaware of the checkpoint | the machine carries on from the moment of creation, with the same open connections and transactions |
| Production | disks after writes are frozen by VSS in a Windows guest or by a file system freeze in a Linux guest | as with a backup | the machine starts from power-off, as after a restore from backup |
A standard checkpoint brings the machine back “mid-flight”, which for a database or a domain controller means returning to a state the rest of the environment knows nothing about. A production checkpoint requires the guest integration services responsible for backup (VSS) to be running. The type is set per machine with Set-VM -CheckpointType: Production creates a production checkpoint and falls back to standard if that fails; ProductionOnly gives a production checkpoint or an error; Standard is always standard; Disabled turns checkpoints off. The default value Production silently creates a standard checkpoint if the production one fails. For database servers and domain controllers ProductionOnly is better: a failed checkpoint shows up as an error, not as a checkpoint of a different type than intended.
Deleting means merging
Deleting a checkpoint in Hyper-V Manager (or with Remove-VMSnapshot) does not remove the file straight away. Hyper-V merges the contents of the AVHDX into the parent disk while the machine runs and only then deletes the file. Merging a file of several hundred gigabytes takes time and puts a heavy load on storage, so a checkpoint kept for months is deleted in a maintenance window.
Domain controllers and backup leftovers
A domain controller running Windows Server 2012 or later on a hypervisor that exposes VM-GenerationID detects that a checkpoint has been applied through the changed identifier and pulls changes from its replication partners, which protects against USN rollback. It is still not a backup: a restored controller receives later deletions from its partners anyway, so it will not bring back a deleted object. The directory backup is a system state backup.
Virtual machine backup software creates a Recovery-type checkpoint during the job and merges it afterwards. An interrupted job may leave it behind, the next job creates another, and after a few weeks the machine has a chain of a dozen AVHDX files that nobody created on purpose. Hyper-V Manager does not always show them, so they are looked for with PowerShell (Get-VMSnapshot -SnapshotType Recovery). A leftover is removed only after making sure no backup job for the machine is in progress and after fixing the cause in the backup system.
When a checkpoint makes sense
A checkpoint is the right tool in one case: before a specific change whose effects can be judged within hours, such as an operating system update, a new application version or a configuration change that might stop the machine from booting. Once the change is verified, the checkpoint is deleted. Applying a checkpoint is one of the ways of rolling back a change. A checkpoint will not survive a storage or host failure, or ransomware with rights on the host; that is the job of a backup on a different system and medium, off-site or offline. It does not replace a backup even before a change: if the change touches data whose loss would be costly, a normal backup is taken first, and the free space on the volume is checked before the checkpoint is created, because every write made during the change will land there.
A checkpoint’s name says who created it, why, and when it is to be deleted. A default name such as “srv-app-01 - (11/08/2026 - 09:14:02)” carries no information a month later. A descriptive name is given by Checkpoint-VM with the -SnapshotName parameter, for example before-update-JS-delete-by-2026-08-13.
Decisions and alternatives
| Approach | When it makes sense | Limitation |
|---|---|---|
| Virtual machine on Hyper-V | full Windows, fixed machine identity, licence per host or core, workloads needed to rebuild the platform, rare releases | slower deployments than containers, no horizontal scaling |
| Container in Kubernetes | stateless applications, frequent releases, horizontal scaling | requires Kubernetes skills in the team; the cluster itself depends on virtual machines |
| Database in the cluster with an operator | the team can rebuild it by hand in a container | the operator requires knowledge of data storage and rehearsed failures |
| Dynamic VHDX disk | many machines on one volume, space allocated on demand | growth beyond usage in the guest; needs TRIM, compaction and free-space monitoring |
| Fixed VHDX disk | full capacity reserved up front | takes the full capacity from the start; needs no compaction |
Production checkpoint (ProductionOnly) | a change on a database server or domain controller | requires working integration services (VSS); a failed checkpoint ends in an error |
| Standard checkpoint | a change where a return with memory state is needed | the application is unaware of the checkpoint; a return to a state unknown to the rest of the environment |
| Backup on a different system and medium | protection against storage failure, host failure and ransomware | a separate cost and restore process; a checkpoint does not replace it |
Baseline configuration
- Workload classification with the three questions (frequency of change, team skills, the platform’s dependence on the workload). The registry, repository, CI and backups stay outside the cluster, with backups on a separate host, so that no service needed to rebuild the cluster depends on the cluster.
- Node machines: generation 2 with the Secure Boot template “Microsoft UEFI Certificate Authority” for Linux, static memory, a fixed number of virtual processors, checkpoints disabled, one time synchronisation variant. Changes on existing nodes:
kubectl drain,Stop-VM, the change,Start-VM,kubectl uncordon, one control plane node at a time; the next node only once the previous one is Ready and all etcd members are present. - A spreading rule for control plane nodes (
AntiAffinityClassNames,ClusterEnforcedAntiAffinityif needed) and a check of actual placement after the groups are restarted. - Linux guest disks: new VHDX disks with a 1 MB block on a SCSI controller, existing ones through
Convert-VHDwith the machine off, no checkpoints and free space for the new file. The old file is deleted after confirming boot from the new disk. In the guestfstrim.timerruns, and the journal, logs and container images have limits. - Checkpoints:
ProductionOnlyfor database servers and domain controllers, and names that carry the author, purpose and deletion date.
Maintenance
-
Inventory of VHDX file sizes against capacity, free space on the host volumes and checkpoints with their age. A
FileSizeclose toSizewith little free space on the volume, and checkpoints older than a few days, signal a problem. In a failover clusterGet-VMneeds-ComputerName (Get-ClusterNode).Name.Get-VM | Get-VMHardDiskDrive | ForEach-Object { Get-VHD -Path $_.Path } | Select-Object Path, VhdType, @{ n = 'SizeGB'; e = { [math]::Round($_.Size / 1GB, 1) } }, @{ n = 'FileSizeGB'; e = { [math]::Round($_.FileSize / 1GB, 1) } } Get-Volume | Where-Object DriveLetter | Select-Object DriveLetter, @{ n = 'FreeGB'; e = { [math]::Round($_.SizeRemaining / 1GB, 1) } } Get-VM | Get-VMSnapshot | Select-Object VMName, Name, SnapshotType, CreationTime, @{ n = 'AgeDays'; e = { [int]((Get-Date) - $_.CreationTime).TotalDays } } | Sort-Object CreationTime -
An alert on free space on the volume holding the machines (e.g. below 15%) and a job reporting the possible growth of non-fixed disks.
-
Old checkpoints deleted in a maintenance window, and disk compaction only after the merge has finished, with a current backup and
FileSizemeasured before and after.
Limitations and pitfalls
- Volume over-commitment. The total capacity of dynamic disks and AVHDX files can exceed the size of the volume, and the problem shows only once the files grow. When the volume runs out of space, Hyper-V pauses (Paused-Critical) the machines that need new blocks, so several machines are affected, not just the one whose disk grew. A free-space alert and regular compaction act before the threshold is crossed.
- Deleting or changing files in the chain by hand. With the machine powered off, deleting the active AVHDX file in File Explorer loses every write since the checkpoint was created, and deleting a file from the middle of the chain makes every file after it useless. Attaching the base VHDX to another machine or editing it breaks the chain, because it changes the parent. The only safe way is to delete the checkpoint through Hyper-V.
- Compacting a disk with checkpoints. The disk of a machine that has checkpoints is not compacted: writes go to the AVHDX, and the base disk is the parent in the chain and must not be modified by any tool. The checkpoints have to be deleted through Hyper-V first and the merge allowed to finish. The operation changes the file in place, takes a long time and loads storage heavily, so the machine should have a current backup beforehand.
- Two time mechanisms at once. The integration service and chrony both setting the clock cause discrepancies, and disabling the integration service without a second time source ends in clock drift. All nodes use one, the same variant.