Failure on arm64

I'm one of the Incus maintainers and as you may know, we've been using virtiofsd as a way to handle writable high speed filesystme sharing with our VMs.

This works as one would expects, the user adds a shared filesystem to a VM, then we will:

  • Spawn virtiofsd (/opt/incus/bin/virtiofsd --fd=3 --cache=never --shared-dir=/var/lib/incus/devices/stgraber-dev_noble-aarch64/disk.foo.foo) where fd #3 (closed) is a unix socket
  • We generate the QEMU configuration which can be seen below
  • We notify our agent in the VM to perform the virtiofs mount

QEMU config:

# foo drive (virtio-fs)
[chardev "incus_foo"]
backend = "socket"
path = "/var/lib/incus/devices/stgraber-dev_noble-aarch64/virtio-fs.foo.sock"

[device "dev-incus_foo-virtio-fs"]
driver = "vhost-user-fs-pci"
bus = "qemu_pcie2"
addr = "00.2"
tag = "incus_foo"
chardev = "incus_foo"

This works great on x86_64 but doing the exact same on aarch64 has been reported by a few different people to be failing.

The VM starts up correctly and the virtiofs export can be seen and mounted, however upon access, virtiofsd on the host immediately dies and guest I/O get stuck.

As far as errors, we're seeing this:

qemu log:

qemu-system-aarch64: Failed to read msg header. Read -1 instead of 12. Original request 0.
qemu-system-aarch64: Failed to write msg. Wrote -1 instead of 20.
qemu-system-aarch64: vhost VQ 1 ring restore failed: -22: Invalid argument (22)
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost VQ 0 ring restore failed: -22: Invalid argument (22)
qemu-system-aarch64: Error starting vhost: 5
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost_set_vring_call failed 22
qemu-system-aarch64: Failed to set msg fds.
qemu-system-aarch64: vhost_set_vring_call failed 22
qemu-system-aarch64: Unexpected end-of-file before all data were read

virtiofsd log:

[2025-07-25T20:14:15Z INFO  virtiofsd] Waiting for vhost-user socket connection...
[2025-07-25T20:14:15Z INFO  virtiofsd] Client connected, servicing requests
[2025-07-25T20:14:30Z ERROR virtiofsd] Waiting for daemon failed: HandleRequest(InvalidParam)

In general, the QEMU setup for both our x86_64 and aarch64 VMs is basically identical. They both use rely on UEFI as firmware (EDK2) and have the same set of PCIe devices at the same addresses. Main difference is that x86_64 uses q35 whereas aarch64 uses virt.

Version wise, we've seen the issue on QEMU 9.0, 9.2 and 10.0 using latest virtiofsd (v1.13.2).