VHDX disks that keep growing: causes, compaction and prevention

A Linux virtual machine running containers has a 200 GB dynamic VHDX disk. In the guest, df -h shows 70 GB in use, while the file on the host is 190 GB and gets bigger every week. The administrator deletes old images and logs in the guest and frees 30 GB, but the VHDX file does not shrink by a single byte. A few months later the volume holding the machines runs out of space, even though according to the guests their data takes up less than half of the disks.

A dynamic VHDX disk grows in one direction only, until the guest tells the host which blocks are free or somebody compacts the file. This post shows where the growth comes from, how to reclaim the space, and how to configure the guest so the problem does not come back.

Three sizes of one disk

A fixed disk takes up its full capacity on the host from the moment it is created and does not grow. A dynamic disk starts as a small file and allocates space in blocks when the guest first writes something to a given area.

With a dynamic disk three figures matter: the capacity seen by the guest, the size of the file on the host, and the data actually in use in the guest. Get-VHD shows the first two:

Get-VHD -Path 'D:\VM\srv-app-01\srv-app-01.vhdx' |
    Select-Object VhdFormat, VhdType, Size, FileSize, MinimumSize, BlockSize

Field	Meaning	Watch out for
`Size`	logical capacity seen by the guest	says nothing about space used on the host
`FileSize`	size of the file on the host volume	on a dynamic disk it never shrinks by itself
`MinimumSize`	smallest capacity `Resize-VHD` can shrink the disk to	follows from the partition layout, not the amount of data
`BlockSize`	allocation unit in the file, 32 MB by default	applies to dynamic and differencing disks
`VhdType`	`Fixed`, `Dynamic` or `Differencing`	`Differencing` is e.g. a checkpoint file

The third figure comes from the guest: df -h on Linux or Get-Volume on Windows. A healthy dynamic disk has a FileSize only slightly larger than the space in use in the guest. A file close to Size while the guest is a third full means that freed space is not returning to the host.

Why the file grows and never shrinks

Hyper-V does not know what the guest keeps in its file system; it sees only writes to sectors. The first write to an area without an allocated block enlarges the file by a whole block. Deleting a file in the guest changes only the file system metadata: the host is never told, so the block stays in the VHDX file.

If the guest always wrote to the same sectors, the file would stop at the level of peak usage. File systems, however, place new data in different parts of the disk, and typical workloads keep creating new files and deleting old ones:

container images: every new application version brings new layers, and old ones are removed only by image clean-up;
logs: rotation creates new files, compresses and deletes old ones;
temporary files, the package manager cache, working directories of CI builds;
updates: downloaded packages, new kernel versions, and on Windows the files of the update service.

After a few months the guest has written something to most areas of the disk and the file approaches full capacity, even though usage in the guest is constant.

Checkpoints are a separate growth mechanism. Once a checkpoint is created, writes go to an AVHDX differencing file, and the space comes back only when the checkpoint is deleted, that is, when the chain is merged. The details are in A checkpoint is not a backup.

Block size: 1 MB for Linux guests

With the default 32 MB block, a 4 KB write to a new area of the disk enlarges the file by 32 MB. In Windows guests this size works well. Linux file systems spread metadata and data across the whole disk (block groups in ext4, allocation groups in XFS), so with a 32 MB block the file grows quickly despite holding little data. Microsoft’s recommendations for Linux guests on Hyper-V specify a dynamic VHDX with a 1 MB block size:

New-VHD -Path 'D:\VM\srv-app-01\srv-app-01-data.vhdx' -SizeBytes 200GB -Dynamic -BlockSizeBytes 1MB

The block size of an existing disk cannot be changed in place. The disk is converted to a new file, which is then attached in place of the old one. Conversion requires the machine to be powered off with no checkpoints, and free space for the new file:

$old = 'D:\VM\srv-app-01\srv-app-01.vhdx'
$new = 'D:\VM\srv-app-01\srv-app-01-1mb.vhdx'

Stop-VM -Name srv-app-01
Convert-VHD -Path $old -DestinationPath $new -VHDType Dynamic -BlockSizeBytes 1MB

# Get-VMHardDiskDrive -VMName srv-app-01 shows the disk's controller and location
Set-VMHardDiskDrive -VMName srv-app-01 -ControllerType SCSI -ControllerNumber 0 -ControllerLocation 0 -Path $new
Start-VM -Name srv-app-01

The old file is deleted only after checking that the machine boots from the new disk and has all its data.

TRIM and UNMAP: the guest reports free blocks

TRIM (ATA) and UNMAP (SCSI) are the commands an operating system uses to tell a disk that a range of sectors no longer holds data. Hyper-V accepts them from the guest, marks the corresponding VHDX blocks as unused and passes the information on to the host storage. Requirements on the machine side:

the disk is in VHDX format; the VHD format does not support UNMAP and has to be converted;
the disk is on a virtual SCSI controller. Generation 2 machines have SCSI controllers only; in generation 1 the system disk sits on an IDE controller.

On a Linux guest, fstrim sends TRIM. In many distributions fstrim.timer is enabled by default and runs once a week; in others it has to be enabled:

# TRIM all mounted file systems that support it, with a report
sudo fstrim -av
# Weekly TRIM run by systemd
sudo systemctl enable --now fstrim.timer

A Windows guest sends delete notifications by default, and the scheduled drive optimisation task repeats TRIM every week. fsutil behavior query DisableDeleteNotify checks the setting (0 means notifications are enabled), and a manual TRIM is run with:

Optimize-Volume -DriveLetter C -ReTrim -Verbose

TRIM stops the growth, because freed blocks can be reused. To shrink the file itself, it has to be compacted.

Compaction: shrinking the file

Optimize-VHD shrinks the file of a dynamic or differencing disk by removing unused blocks; it does not work on fixed disks. The disk must not be in use by a running machine, and in Quick and Full modes it must be attached to the host read-only. Full mode also looks for blocks filled with zeros.

How effective it is depends on the guest file system. When the disk holds NTFS, the host can read free-space information from it. Windows cannot read ext4 or XFS, so with a Linux guest compaction reclaims the blocks the guest has already marked with TRIM and blocks filled with zeros. That is why sudo fstrim -av is run on a Linux machine just before it is shut down.

$vm   = 'srv-app-01'
$vhdx = 'D:\VM\srv-app-01\srv-app-01.vhdx'

Stop-VM -Name $vm                       # shut down the guest operating system
$before = (Get-VHD -Path $vhdx).FileSize
Mount-VHD -Path $vhdx -ReadOnly         # Full mode requires a read-only attachment
Optimize-VHD -Path $vhdx -Mode Full     # removes unused and zeroed blocks
Dismount-VHD -Path $vhdx
Start-VM -Name $vm
'{0:N1} GB -> {1:N1} GB' -f ($before / 1GB), ((Get-VHD -Path $vhdx).FileSize / 1GB)

Prevention

Compaction reclaims space once. To keep the file from returning to its previous size, the guest has to send TRIM regularly and must not accumulate, without limits, data that will be deleted anyway.

System journal. By default journald uses up to 10% of the file system, capped at 4 GB. A lower limit is set in a drop-in file; the change takes effect after sudo systemctl restart systemd-journald, and journalctl --disk-usage shows the current size:

[Journal]
SystemMaxUse=1G

Application logs. Applications that write to their own files need a logrotate rule limiting the number and size of files:

/var/log/app/*.log {
    daily
    rotate 14
    maxsize 200M
    compress
    delaycompress
    missingok
    notifempty
    # for applications that cannot reopen their log file
    copytruncate
}

Container images. The kubelet removes unused images when usage of the image file system exceeds imageGCHighThresholdPercent (85% by default) and cleans up down to imageGCLowThresholdPercent (80% by default). Lower thresholds leave fewer old layers on the disk. In k3s they are set as kubelet arguments:

kubelet-arg:
  - "image-gc-high-threshold=75"
  - "image-gc-low-threshold=60"

On image build hosts, unused images are removed by a scheduled podman image prune -a or docker image prune -a, and on a Kubernetes node a one-off crictl rmi --prune does the same.

Monitoring. The minimum is an alert on free space on the volume holding the machines, e.g. below 15%. An earlier signal comes from the second script in “Check your own environment”, run from Task Scheduler and sending the possible growth of disks on each volume to monitoring. A file whose FileSize grows every week while usage in the guest stays flat points to a machine without working TRIM.

Symptom, cause, action

Symptom	Cause	What to do
`FileSize` close to `Size` while `df -h` in the guest shows 30–40%	the guest does not send TRIM	enable `fstrim.timer`, run `fstrim -av`, compact
`lsblk --discard` shows 0 in DISC-GRAN and DISC-MAX	VHD format or an IDE controller	convert to VHDX, move data disks to a SCSI controller
A new Linux guest with a few GB of data has a file many times larger	32 MB block size	new disk with a 1 MB block size or `Convert-VHD`
`.avhdx` files growing next to the VHDX	a checkpoint or a leftover from a backup job	delete the checkpoint through Hyper-V, never remove files by hand
`Optimize-VHD` reclaims almost nothing with a Linux guest	the host cannot read ext4 or XFS	run `fstrim -av` before shutting the machine down, then compact

Check your own environment

The disks of all machines on the host with their type, controller, block size and the ratio of file size to capacity:

Get-VM | Get-VMHardDiskDrive | ForEach-Object {
    $vhd = Get-VHD -Path $_.Path
    [pscustomobject]@{
        VM          = $_.VMName
        Controller  = $_.ControllerType
        Type        = $vhd.VhdType
        BlockMB     = $vhd.BlockSize / 1MB
        SizeGB      = [math]::Round($vhd.Size / 1GB)
        FileGB      = [math]::Round($vhd.FileSize / 1GB, 1)
        FilePercent = [math]::Round(100 * $vhd.FileSize / $vhd.Size)
    }
} | Sort-Object FilePercent -Descending | Format-Table -AutoSize

A good result: Linux guest disks with a 1 MB block size, on a SCSI controller, with FilePercent close to usage in the guest. A bad result: FilePercent above 90 with the guest half full, BlockMB of 32 on a Linux guest, or a data disk on IDE. A Differencing row is a checkpoint, and its FileGB is the size of the last file in the chain only.

Free space on the volumes and how much the dynamic disks can still grow. The script assumes volumes with a drive letter; for Cluster Shared Volumes it needs adapting:

Get-VM | Get-VMHardDiskDrive | ForEach-Object { Get-VHD -Path $_.Path } |
    Group-Object { (Split-Path $_.Path -Qualifier).TrimEnd(':') } |
    ForEach-Object {
        $vol    = Get-Volume -DriveLetter $_.Name
        $growth = ($_.Group | Where-Object VhdType -ne 'Fixed' |
                   ForEach-Object { $_.Size - $_.FileSize } | Measure-Object -Sum).Sum
        [pscustomobject]@{
            Volume           = $_.Name
            FreeGB           = [math]::Round($vol.SizeRemaining / 1GB)
            FreePercent      = [math]::Round(100 * $vol.SizeRemaining / $vol.Size)
            PossibleGrowthGB = [math]::Round($growth / 1GB)
        }
    } | Format-Table -AutoSize

A good result: FreeGB greater than PossibleGrowthGB, or a deliberate overcommitment with an alert in place. A bad result: FreePercent below 15, or possible growth several times larger than the free space with no alert.

On a Linux guest: whether the disks accept TRIM, whether the timer runs, and what the actual usage is:

lsblk --discard                    # DISC-GRAN and DISC-MAX above 0 = the disk accepts TRIM
systemctl status fstrim.timer      # expected state: active (waiting)
systemctl list-timers fstrim.timer # last and next run
df -h                              # file system usage in the guest

A good result: non-zero DISC-GRAN and DISC-MAX, the timer active (waiting) with its last run within the past week, and usage from df -h close to FilePercent. A bad result: zeros in the DISC columns, the timer inactive or unknown, usage in the guest clearly lower than FilePercent.