Interactive Troubleshooter

Published: Sep 17, 2026

Proxmox VM 100% IO Delay: ZFS and Disk Diagnostic

Time: 15 mins Level: Beginner

By StepDiag Engineering Team

Reviewed by Ibraheem Taofeeq Opeyemi

An exhaustive root-cause diagnostic tree for diagnosing and resolving severe Proxmox VE IO delay spikes and VM freezing. Covers ZFS ARC pressure, QEMU block layer cache misconfigurations, consumer SSD write amplification, and ZFS pool degraded states.

Full Diagnostic Tree & Step-by-Step Overview

What primary symptom or storage metric behavior correlates with your Proxmox IO delay spike?

System-wide IO delay reaches 50-100% on ZFS pools during sync write operations or high RAM usage
Single VM stalls or freezes under heavy disk activity while host storage utilization appears moderate
Consumer-grade SSDs or NVMe drives experience severe latency degradation over time
ZFS Pool reports degraded state, high scrub activity, or continuous checksum errors

Identify your specific ZFS pool configuration or resource constraint on the Proxmox host:

ZFS Adaptive Replacement Cache (ARC) is consuming host RAM, triggering swap file thrashing
ZFS synchronous writes (sync=always or databases) forcing extreme disk write latency
Incorrect ZFS pool ashift value causing write amplification on 4K/Advanced Format drives
ZFS auto-trim operations triggering massive IO blocking spikes on SSD arrays

Cap ZFS Adaptive Replacement Cache (ARC) and Reclaim Host Memory

Solution:

Root Cause: ZFS ARC Exhausting Host RAM and Inducing Linux Kernel Swap Thrashing

By default, ZFS on Linux consumes up to 50% of total system memory for its Adaptive Replacement Cache (ARC). On Proxmox VE nodes with heavily provisioned VMs, the Linux kernel's out-of-memory avoidance mechanisms force pages into swap space when KVM guest requests memory faster than ZFS can shrink its ARC. This results in severe kernel iowait (IO Delay) as virtual machine memory ranges are continuously read from and written to slow swap storage.

# Diagnostic Verification:

Execute the following terminal commands on the Proxmox host to verify memory utilization and ZFS ARC size:

bash
free -h
arcstat
swapon --show

Observe if arcstat shows ARC memory near maximum while swap usage (swapon) is actively increasing during IO delay spikes.

# Step-by-Step Fix:

1. Limit Maximum ZFS ARC Size:

Edit or create the ZFS module configuration file in /etc/modprobe.d/zfs.conf:

bash
     nano /etc/modprobe.d/zfs.conf

Set zfs_arc_max to a hard ceiling (e.g., 16 GB = 17179869184 bytes for a 64 GB host):

text
     options zfs zfs_arc_max=17179869184
     options zfs zfs_arc_min=8589934592

2. Apply Module Configuration and Update Initramfs:

Apply the new ZFS parameter dynamically without rebooting:

bash
     echo 17179869184 > /sys/module/zfs/parameters/zfs_arc_max

Persist changes across host reboots by updating the initial RAM disk:

bash
     update-initramfs -u -k all

3. Tune Kernel Swappiness:

Reduce kernel aggressiveness when swapping host memory pages by setting vm.swappiness in /etc/sysctl.conf:

bash
     sysctl -w vm.swappiness=10
     echo "vm.swappiness=10" >> /etc/sysctl.conf

# Prevention & Long-Term Monitoring:

Ensure the sum of maximum guest VM RAM allocation + ZFS zfs_arc_max does not exceed 85% of total physical host memory.

Configure ZFS Sync Writes and Dedicated SLOG Device

Solution:

Root Cause: Synchronous IO Bottlenecks (ZFS ZIL Blocking Transaction Groups)

Applications like PostgreSQL, MySQL, and Proxmox HA cluster logging call synchronous write operations (fsync/O_SYNC). ZFS forces synchronous writes to be committed to the ZFS Intent Log (ZIL) before acknowledging completion to the VM. When the ZIL resides on standard pool vdevs without power-loss-protected high-write-IOPS storage, disk write amplification triggers near 100% IO delay as transaction groups (TXGs) stall.

# Diagnostic Verification:

Monitor real-time synchronous write requests and latency across pool vdevs:

bash
zpool iostat -v 1
zfs get sync <pool_name>

Check if disk latency spikes specifically during heavy database or VM journal writes.

# Step-by-Step Fix:

1. Option A: Add a Power-Loss Protected (PLP) Dedicated SLOG Device:

Attach an enterprise NVMe SSD featuring Power Loss Protection (e.g., Intel Optane or Kioxia CD series) as a dedicated ZIL device:

bash
     zpool add <pool_name> log /dev/disk/by-id/nvme-ENTERPRISE_SSD_ID

2. Option B: Adjust ZFS Pool Sync Property (Use with Caution):

For non-critical workloads, lab environments, or systems backed by UPS power, override synchronous enforcement on specific datasets:

bash
     zfs set sync=disabled <pool_name>/<dataset_name>

*Warning:* Disabling sync introduces a window of potential data loss (up to 5 seconds of uncommitted writes) in the event of an abrupt host power failure.

# Prevention & Long-Term Monitoring:

Reference Proxmox Official ZFS Documentation for recommended dataset configuration and hardware layout guidelines.

Correct Misaligned ZFS Pool Ashift Settings

Solution:

Root Cause: Misaligned ZFS Block Size (Ashift) and Physical Sector Amplification

If a ZFS pool was created with ashift=9 (512-byte block size alignment) on modern drives utilizing 4096-byte (4K) physical sectors or 8K/16K NVMe block structures, every single write operation forces a read-modify-write cycle across the physical drive controller. This produces up to an 8x write amplification factor, swamping drive queues and causing host IO delay to hover near 100% under moderate guest IOPS.

# Diagnostic Verification:

Inspect the active ashift value configured on your ZFS pool vdevs:

bash
zpool get all <pool_name> | grep ashift

If ashift reports 9 on modern Advanced Format (4Kn/512e) HDDs or SSDs, the pool is misaligned.

# Step-by-Step Fix:

1. Determine Native Physical Sector Size:

Check the underlying physical drive block sizes:

bash
     smartctl -a /dev/sda | grep -i "Sector Size"
     lsblk -o NAME,PHY-SEC,LOG-SEC

2. Re-create ZFS Pool with Optimal Alignment (ashift=12 or ashift=13):

Note: ashift cannot be modified on an existing live vdev; data must be backed up via Proxmox Backup Server or zfs send and restored to a rebuilt pool.

Export workloads and create the pool enforcing 4K (ashift=12) or 8K (ashift=13) alignment:

bash
     zpool create -f -o ashift=12 -O compression=on -O acltype=posixacl <pool_name> mirror /dev/sdA /dev/sdB

# Prevention & Long-Term Monitoring:

Always explicitly specify -o ashift=12 when building ZFS pools through CLI, or select ashift=12 in the Proxmox GUI installer.

Tune ZFS Auto-Trim and Asynchronous Trim Parameters

Solution:

Root Cause: Synchronous Auto-Trim Saturating SSD Command Queues

Enabling autotrim=on on ZFS pools hosted on non-enterprise SSDs or drives lacking high-performance queue processing forces the drive controller to execute TRIM (Dataset Management) commands inline with standard read/write queues. This causes severe queue head-of-line blocking, resulting in temporary 100% IO delay spikes every time blocks are unlinked.

# Diagnostic Verification:

Verify if ZFS autotrim is active on the pool:

bash
zpool get autotrim <pool_name>

Check system logs for ATA/NVMe queue reset errors or latency spikes using dmesg -T | grep -i trim.

# Step-by-Step Fix:

1. Disable Real-Time Auto-Trim on the ZFS Pool:

Turn off inline auto-trim enforcement:

bash
     zpool set autotrim=off <pool_name>

2. Schedule Asynchronous Periodic Trims via Systemd Timer:

Utilize periodic background trimming during low-load off-peak hours:

bash
     systemctl enable --now zfs-trim.timer

Alternatively, execute a manual background trim manually:

bash
     zpool trim <pool_name>

# Prevention & Long-Term Monitoring:

Monitor trim progress using zpool status -t <pool_name> to verify completion without impacting production guest workloads.

Identify the storage driver, cache mode, or disk image format configured for the affected VM:

VM disk controller set to IDE or VirtIO Block without IOThreads enabled
VM disk caching mode configured to Default (No Cache) under heavy guest write queues
QEMU Async IO backend using default io=threads instead of io=io_uring
QCOW2 disk image file fragmentation or missing discard/trim pass-through flags

Migrate Virtual Disk to VirtIO-SCSI Single with Dedicated IOThreads

Solution:

Root Cause: Single-Threaded QEMU Event Loop Bottlenecking Guest Disk I/O

By default, Proxmox routes virtual machine storage traffic through the main QEMU process thread. When a guest OS generates significant IOPS, disk processing competes directly with virtual CPU execution and display emulation on that single CPU thread, causing high VM-level IO wait times and unresponsive guest applications.

# Diagnostic Verification:

Check the VM hardware configuration file located in /etc/pve/qemu-server/<vmid>.conf:

bash
qm config <vmid>

Look for disk definitions using ideX, sataX, or scsiX missing the iothread=1 parameter.

# Step-by-Step Fix:

1. Change SCSI Controller to VirtIO-SCSI Single:

Stop the target VM (qm stop <vmid>).

In the Proxmox Web UI, navigate to VM > Hardware > Options > SCSI Controller and select VirtIO SCSI single.

Or via CLI:

bash
     qm set <vmid> --scsihw virtio-scsi-single

2. Enable IOThread on Virtual Disks:

Edit the disk interface to enable dedicated IO processing threads:

bash
     qm set <vmid> --scsi0 <storage_name>:vm-<vmid>-disk-0,iothread=1

3. Start the VM and Confirm Thread Allocation:

Start the VM and check running QEMU thread tasks via htop or ps -T -p $(pgrep qemu-system) to confirm isolated disk handling threads.

# Prevention & Long-Term Monitoring:

Ensure all high-IOPS virtual machines (database servers, file servers) are deployed using virtio-scsi-single with iothread=1 as standard operating policy.

Optimize VM Cache Settings (Write Back vs. Direct Sync)

Solution:

Root Cause: Uncached Disk I/O Forcing Synchronous Host Subsystem Serialization

When a virtual machine disk cache is set to No cache (cache=none), the guest kernel bypasses host page caching entirely. While this ensures safety across live migration, non-volatile underlying storage without power-loss protection forces write requests to wait for complete physical media acknowledgement before completing, bottlenecking guest tasks.

# Diagnostic Verification:

Inspect the active disk cache setting in /etc/pve/qemu-server/<vmid>.conf:

bash
grep cache /etc/pve/qemu-server/<vmid>.conf

# Step-by-Step Fix:

1. Configure Optimal Cache Mode based on Infrastructure:

Write Back (cache=writeback): Offers maximum performance by utilizing the host page cache. Ideal for workloads where high IOPS are critical and the host is connected to an Uninterruptible Power Supply (UPS).

Write Through (cache=writethrough): Reads are cached, but writes complete only when media accepts data. Safe, but slower write performance.

2. Apply Write Back Cache via Proxmox CLI:

Execute the following command to update the VM disk definition:

bash
     qm set <vmid> --scsi0 <storage_name>:vm-<vmid>-disk-0,cache=writeback

3. Reboot Virtual Machine:

A full stop and start of the VM is required for QEMU block layer cache flags to take effect.

# Prevention & Long-Term Monitoring:

Balance cache selection based on host resilience—use writeback for high-throughput nodes backed by dual-pdu hardware and battery backup.

Switch QEMU Async IO Engine to io_uring

Solution:

Root Cause: High Syscall Overhead with Legacy Async IO (threads/aio=native)

Proxmox QEMU utilizes asynchronous IO engines to process disk requests. Legacy modes like aio=threads rely on POSIX thread pools, incurring context-switch overhead under high IOPS. Mode aio=native relies on Linux io_submit, which can stall on ZFS file-backed storage or raw devices during metadata updates.

# Diagnostic Verification:

Check the current aio engine configured for the VM drive:

bash
qm config <vmid> | grep aio

# Step-by-Step Fix:

1. Set Async IO Engine to io_uring:

Modern Linux kernels included in Proxmox VE support io_uring, which drastically minimizes system call overhead by utilizing shared submission/completion ring buffers:

bash
     qm set <vmid> --scsi0 <storage_name>:vm-<vmid>-disk-0,aio=io_uring

2. Verify Linux Kernel Support:

Confirm the host is running Proxmox kernel 5.15 or newer (PVE 7.x/8.x) where io_uring stability is production-ready:

bash
     uname -r

# Prevention & Long-Term Monitoring:

Adopt aio=io_uring as the default asynchronous IO setting across all Linux-based guest templates.

Enable Discard/TRIM Pass-through and Defragment QCOW2 Storage

Solution:

Root Cause: Unused Block Accumulation and QCOW2 Image Metadata Bloat

When guest operating systems delete files inside a virtual disk without TRIM/Discard enabled, the underlying storage layer (ZFS sparse volumes or QCOW2 files) cannot reclaim those freed blocks. Over time, QCOW2 files experience extreme metadata fragmentation, forcing the host to execute multiple physical read/write operations for single guest file accesses.

# Diagnostic Verification:

Compare reported guest file system usage versus host volume block consumption:

bash
df -h # Inside Guest VM
qemu-img info /var/lib/vz/images/<vmid>/vm-<vmid>-disk-0.qcow2 # On Host

# Step-by-Step Fix:

1. Enable Discard Flag on Virtual Disk:

Configure Proxmox to pass guest discard commands through to underlying storage:

bash
     qm set <vmid> --scsi0 <storage_name>:vm-<vmid>-disk-0,discard=on

2. Run Guest File System Trim:

Inside the guest Linux virtual machine, execute fstrim:

bash
     sudo fstrim -va

3. Convert/Shrink Fragmented QCOW2 Images:

Offline compact QCOW2 storage files to remove unallocated fragmentation blocks:

bash
     qemu-img convert -O qcow2 -c /path/to/old_disk.qcow2 /path/to/compact_disk.qcow2

# Prevention & Long-Term Monitoring:

Enable fstrim.timer in all guest Linux OS installations to ensure continuous block reclamation.

Identify physical SSD controller characteristics or performance symptoms:

Consumer SSD lacking Power Loss Protection (PLP) experiencing cache exhaustion
High SSD Wearout indicator or SMART write amplification failure
NVMe thermal throttling or PCIe bus power management state latency

Replace Consumer Drives / Configure Workaround for Non-PLP SSDs

Solution:

Root Cause: DRAM-less or Consumer SSD SLC Cache Exhaustion under ZFS Workloads

Consumer-grade SSDs (e.g., Samsung EVO, Crucial MX/BX series) rely on small SLC cache buffers and volatile DRAM. ZFS continuously writes copy-on-write metadata and transaction groups. Once the consumer drive's SLC cache fills up (often within 10–30 seconds of sustained write activity), drive write speeds collapse from 500 MB/s to as low as 20–50 MB/s—far lower than a traditional mechanical hard drive. This saturates host wait queues and spikes IO delay to 100%.

# Diagnostic Verification:

Run iostat while generating test write activity on the storage pool:

bash
iostat -xz 1

Observe if %util approaches 100% while wMB/s drops drastically and await (latency) climbs into hundreds of milliseconds.

# Step-by-Step Fix:

1. Immediate Temporary Workaround (Reduce ZFS TXG Flush Size):

Reduce the ZFS transaction group dirty data limit to prevent overwhelming consumer drive cache buffers:

bash
     echo 1073741824 > /sys/module/zfs/parameters/zfs_dirty_data_max

2. Long-Term Hardware Mitigation:

Replace consumer drives with Enterprise-grade SAS/SATA/NVMe SSDs that feature dedicated Power Loss Protection (PLP) and high sustained write endurance (DWPD >= 1-3).

Consult vendor specifications to verify continuous random 4K write IOPS ratings.

# Prevention & Long-Term Monitoring:

Never deploy DRAM-less consumer SSDs in enterprise Proxmox ZFS storage pools.

Diagnose High SSD Wearout and Replace Failing Storage Media

Solution:

Root Cause: Flash Cell Degradation and Internal Controller Error Recovery Latency

As solid-state drives approach their total bytes written (TBW) limits, internal flash memory cells degrade. The drive controller spends excessive time performing error correction code (ECC) recovery, bad block management, and internal garbage collection. This causes random severe read/write latency spikes on the host node.

# Diagnostic Verification:

Query SMART metrics and health attributes across all installed storage drives:

bash
smartctl -a /dev/nvme0n1
pveversion
lsblk

Look for Percentage Used, Media and Data Integrity Errors, or SMART attribute 233 Media_Wearout_Indicator dropping near 0.

# Step-by-Step Fix:

1. Identify Degrading Drive:

Run SMART self-tests to identify drives reporting uncorrectable sector or read errors:

bash
     smartctl -t short /dev/sda
     smartctl -l error /dev/sda

2. Perform Safe Hot-Swap or Pool Replacement:

If the drive is part of a ZFS pool, offline the degrading device and execute a replace command:

bash
     zpool offline <pool_name> /dev/disk/by-id/OLD_DRIVE_ID
     zpool replace <pool_name> /dev/disk/by-id/OLD_DRIVE_ID /dev/disk/by-id/NEW_DRIVE_ID

3. Monitor Resilver Process:

Track reconstruction progress using zpool status 1.

# Prevention & Long-Term Monitoring:

Configure Proxmox smartd alerts or integrate with Prometheus/Grafana to receive notifications when drive wearout exceeds 80%.

Mitigate NVMe Thermal Throttling and PCIe ASPM Latency

Solution:

Root Cause: NVMe Thermal Throttling or PCIe Active State Power Management (ASPM) Delays

High-performance NVMe SSDs under sustained virtualization workloads generate substantial heat. When controller temperatures exceed thermal thresholds (typically 70°C–85°C), the drive drops into severe thermal throttling power states, dropping throughput to protect silicon. Alternatively, aggressive host PCIe power saving modes (ASPM) introduce state transition delays every time a sleeping bus is woken by VM I/O.

# Diagnostic Verification:

Check NVMe temperature sensors and thermal throttle log counters:

bash
nvme smart-log /dev/nvme0 | grep -E "temperature|warning|critical_warning"

Check kernel power management settings via dmesg | grep -i aspm.

# Step-by-Step Fix:

1. Disable PCIe ASPM Power Savings in Kernel Boot Parameters:

Edit /etc/default/grub to append pcie_aspm=off to GRUB_CMDLINE_LINUX_DEFAULT:

text
     GRUB_CMDLINE_LINUX_DEFAULT="quiet console=tty0 pcie_aspm=off"

Update GRUB bootloader configuration:

bash
     update-grub

*(For systemd-boot installations, add pcie_aspm=off to /etc/kernel/cmdline and execute proxmox-boot-tool refresh.)*

2. Improve Physical Server Cooling / Heatsinks:

Ensure NVMe M.2 drives are fitted with dedicated heatsinks and server chassis fans maintain high linear airflow over the PCIe slots.

# Prevention & Long-Term Monitoring:

Monitor host hardware thermals continuously using sensors or ipmitool daemon integration.

Select the ZFS pool status or fault condition observed on the host:

ZFS Pool is in DEGRADED state due to a failed drive or disconnected bus cable
Background ZFS Scrub operation is consuming 100% storage bandwidth during production hours

Resolve ZFS Pool Degraded State and Rebuild Vdev

Solution:

Root Cause: Degraded RAID Mirror/RAIDZ Pool Operating in Fault-Tolerant Reconstruct Mode

When a physical drive in a ZFS vdev fails or suffers severe connection dropouts, the ZFS pool enters a DEGRADED state. Every subsequent read operation targeting missing data blocks forces ZFS to dynamically calculate and reconstruct data on-the-fly using parity or mirror copies from remaining drives. This parity computation overhead coupled with asymmetric disk reads causes overall IO delay to skyrocket.

# Diagnostic Verification:

Check ZFS pool state and pinpoint the specific faulted device node:

bash
zpool status -v

Verify disk hardware link status using dmesg -T | grep -E "ata|scsi|nvme|reset".

# Step-by-Step Fix:

1. Locate and Replace the Defective Physical Drive:

Identify drive bay location using serial numbers via smartctl -i /dev/sdX.

Unmount/offline the device in ZFS if still connected:

bash
     zpool offline <pool_name> /dev/disk/by-id/FAILED_DRIVE_ID

2. Clear Bus Errors or Hot-Swap Replacement Drive:

Swap the faulty disk with a healthy drive of equal or greater capacity.

3. Initiate ZFS Resilver Operation:

Trigger the pool rebuild process:

bash
     zpool replace <pool_name> /dev/disk/by-id/FAILED_DRIVE_ID /dev/disk/by-id/NEW_DRIVE_ID

4. Monitor Resilvering Latency:

Monitor progress until pool status returns to ONLINE with zero errors:

bash
     zpool status <pool_name>

# Prevention & Long-Term Monitoring:

Configure automated email alerts in /etc/aliases on the Proxmox node to catch ZFS pool state changes immediately.

Throttle ZFS Scrub Priority and Reschedule Maintenance

Solution:

Root Cause: High-Priority Background ZFS Scrub Saturated Storage Queue

ZFS scrubs systematically read every data block in the storage pool to verify checksum integrity. When a scrub executes automatically during peak production hours, default kernel zfs task limits allow the scrub process to contend heavily with guest virtual machine read/write queues, elevating IO delay to 100%.

# Diagnostic Verification:

Check if a scrub is currently active on the pool:

bash
zpool status <pool_name> | grep scrub

# Step-by-Step Fix:

1. Pause or Cancel Active Production Scrub:

If IO delay is impacting live operations, temporarily pause the scrub:

bash
     zpool scrub -s <pool_name>

2. Throttle ZFS Scrub/Resilver IO Limits Dynamically:

Reduce the maximum asynchronous I/O requests permitted for scrub operations in system memory:

bash
     echo 1 > /sys/module/zfs/parameters/zfs_scan_max_active_tasks
     echo 1000 > /sys/module/zfs/parameters/zfs_scrub_delay

3. Adjust Cron/Systemd Scrub Schedules:

Re-schedule automated pool scrubs (/etc/cron.d/zfsutils-linux or systemd zfs-scrub.timer) to execute strictly on weekends or off-peak hours.

# Prevention & Long-Term Monitoring:

Keep ZFS scrub delay parameters tuned permanently via /etc/modprobe.d/zfs.conf to guarantee guest VM IO priority over background maintenance tasks.

Related Diagnostic Guides

Let's Encrypt Rate Limit Hit: Bypass and Wait Strategy

Node.js ERR_MODULE_NOT_FOUND After Move to ESM

Python Virtual Environment Broken After Python Upgrade

Docker Build Context Too Large: .dockerignore Diagnostic

VS Code Remote SSH Keeps Disconnecting: KeepAlive Fix

Subscribe to StepDiag

Embed Guide on Your Website