USB Passthrough Fails on Start, Needs domain Reset Hi, I am seeing (consistently = always), USB Passthrough for my Logitech Keyboard and Mouse ... they don't work / no response on domain (VM) startup. After a reset of the VM they then work - but why are they "dead" on initial startup of the VM? Is this a known issue? Running, QEMU emulator version 5.0.0 (Debian 1:5.0-5ubuntu9.1) Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers And if it makes a difference, this is a macOS guest (on a Linux host). Thanks! Hi Russel, I haven't seen such issues recently, but let us try to sort it out. For that I beg your pardon but I need to start asking for a few details: 1. what exactly (commands) does "reset of the VM" mean for you? 2. in the guest does the output of lspci -v (or whatever the macos counterpart is) change before/after reset and if so how does it change? 3. Could you track on your host "journalctl -f" output, then start the guest, then reset the guest - and attach that log here. If possible please identify the timestamps when you have reset the guest. Sure, NP at all! Let me try to answer, and yell if you need more details - I will get them for you. 1) Reset ... meaning if I start the domain with 'virsh start macOS' => no USB passthrough (Logitech nano receiver in this case). So then, I 'virsh reset macOS' ... USB passthrough works fine! But always needs the reset after start. Make sense? 2) Can't really check that, as I never get in to the guest OS. I can't even get into the UEFI, given that the passthrough device is a receiver for a keyboard and mouse. So it's right at the start where it's not available ... or better said, it's not available / working right from the start. 3) Sure, NP at all! Do you want me to filter journalctl, to only capture a specific target (e.g. libvirtd)? Thanks! (1) yeah makes sense now, thanks (2) So we can't get good-vs-bad data, but is the behavior any different at least with non MacOS guests then? (3) no please no filtering - add all of it to see everything from kernel-apparmor denials to anything odd with udev rules. (2)+(3) bonus, since the device will need to be detached on the host to be passed through and IIRC that depends a bit on USB topology. Can you provide pre/booted/after-reset output of "lsusb -t" of the host as well pelase? Sure, will add - NP! One comment on (2) ... I can't say if this happens with other guests or not, as I only bind this USB device to this particular (guest) OS. OK, let me try to explain what I did here - and yell if this is not clear! Step #1, check lsusb -t before any qemu start, /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 3: Dev 26, If 0, Class=Human Interface Device, Driver=usbfs, 1.5M |__ Port 4: Dev 3, If 0, Class=Communications, Driver=rndis_host, 480M |__ Port 4: Dev 3, If 1, Class=CDC Data, Driver=rndis_host, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M |__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=uas, 10000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M |__ Port 1: Dev 4, If 2, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 1: Dev 4, If 0, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 1: Dev 4, If 1, Class=Human Interface Device, Driver=usbhid, 12M |__ Port 7: Dev 3, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M |__ Port 10: Dev 5, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M Step #2, virsh start macOS, then I have, a) lsusb -t /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 3: Dev 26, If 0, Class=Human Interface Device, Driver=usbfs, 1.5M |__ Port 4: Dev 3, If 0, Class=Communications, Driver=rndis_host, 480M |__ Port 4: Dev 3, If 1, Class=CDC Data, Driver=rndis_host, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M |__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=uas, 10000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M |__ Port 1: Dev 4, If 2, Class=Human Interface Device, Driver=usbfs, 12M |__ Port 1: Dev 4, If 0, Class=Human Interface Device, Driver=usbfs, 12M |__ Port 1: Dev 4, If 1, Class=Human Interface Device, Driver=usbfs, 12M |__ Port 7: Dev 3, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M |__ Port 10: Dev 5, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M b) Output from, journalctl -f | grep -v x11vnc (avoid x11vnc flooding), Dec 01 14:35:32 linuxServer audit[845332]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845332 comm="apparmor_parser" Dec 01 14:35:32 linuxServer kernel: audit: type=1400 audit(1606854932.259:190): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845332 comm="apparmor_parser" Dec 01 14:35:32 linuxServer audit[845341]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845341 comm="apparmor_parser" Dec 01 14:35:32 linuxServer kernel: audit: type=1400 audit(1606854932.451:191): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845341 comm="apparmor_parser" Dec 01 14:35:32 linuxServer audit[845345]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845345 comm="apparmor_parser" Dec 01 14:35:32 linuxServer kernel: audit: type=1400 audit(1606854932.643:192): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845345 comm="apparmor_parser" Dec 01 14:35:32 linuxServer audit[845352]: AVC apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845352 comm="apparmor_parser" Dec 01 14:35:32 linuxServer kernel: audit: type=1400 audit(1606854932.839:193): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845352 comm="apparmor_parser" Dec 01 14:35:32 linuxServer kernel: virbr0: port 2(vnet0) entered blocking state Dec 01 14:35:32 linuxServer kernel: virbr0: port 2(vnet0) entered disabled state Dec 01 14:35:32 linuxServer kernel: device vnet0 entered promiscuous mode Dec 01 14:35:32 linuxServer kernel: virbr0: port 2(vnet0) entered blocking state Dec 01 14:35:32 linuxServer kernel: virbr0: port 2(vnet0) entered listening state Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8545] manager: (vnet0): new Tun device (/org/freedesktop/NetworkManager/Devices/13) Dec 01 14:35:32 linuxServer systemd-udevd[845358]: Using default interface naming scheme 'v245'. Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8690] device (vnet0): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external') Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8710] device (vnet0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external') Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8716] device (vnet0): Activation: starting connection 'vnet0' (1e7b8ae9-ce83-48fd-9381-38a14aac986a) Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8718] device (vnet0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external') Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8722] device (vnet0): state change: prepare -> config (reason 'none', sys-iface-state: 'external') Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8725] device (vnet0): state change: config -> ip-config (reason 'none', sys-iface-state: 'external') Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8726] device (virbr0): bridge port vnet0 was attached Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8727] device (vnet0): Activation: connection 'vnet0' enslaved, continuing activation Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8728] device (vnet0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external') Dec 01 14:35:32 linuxServer dbus-daemon[1647]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.9' (uid=0 pid=1648 comm="/usr/sbin/NetworkManager --no-daemon ") Dec 01 14:35:32 linuxServer systemd[1]: Starting Network Manager Script Dispatcher Service... Dec 01 14:35:32 linuxServer dbus-daemon[1647]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher' Dec 01 14:35:32 linuxServer systemd[1]: Started Network Manager Script Dispatcher Service. Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8933] device (vnet0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external') Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8935] device (vnet0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external') Dec 01 14:35:32 linuxServer NetworkManager[1648]: [1606854932.8945] device (vnet0): Activation: successful, device activated. Dec 01 14:35:32 linuxServer systemd[1]: Reloading Postfix Mail Transport Agent (instance -). Dec 01 14:35:32 linuxServer postfix/postfix-script[845421]: refreshing the Postfix mail system Dec 01 14:35:32 linuxServer postfix/master[5209]: reload -- version 3.5.6, configuration /etc/postfix Dec 01 14:35:32 linuxServer systemd[1]: Reloaded Postfix Mail Transport Agent (instance -). Dec 01 14:35:32 linuxServer systemd[1]: Reloading Postfix Mail Transport Agent. Dec 01 14:35:32 linuxServer systemd[1]: Reloaded Postfix Mail Transport Agent. Dec 01 14:35:32 linuxServer nm-dispatcher[845429]: /etc/network/if-up.d/resolved: 12: mystatedir: not found Dec 01 14:35:33 linuxServer audit[845373]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845373 comm="apparmor_parser" Dec 01 14:35:33 linuxServer kernel: audit: type=1400 audit(1606854933.051:194): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-d06d502a-904a-4b34-847d-debf1a3d76c7" pid=845373 comm="apparmor_parser" Dec 01 14:35:33 linuxServer libvirtd[1678598]: Domain id=4 name='macOS' uuid=d06d502a-904a-4b34-847d-debf1a3d76c7 is tainted: custom-argv Dec 01 14:35:33 linuxServer systemd-machined[1695]: New machine qemu-4-macOS. Dec 01 14:35:33 linuxServer systemd[1]: Started Virtual Machine qemu-4-macOS. Dec 01 14:35:33 linuxServer avahi-daemon[1646]: Joining mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe92:d47b. Dec 01 14:35:33 linuxServer avahi-daemon[1646]: New relevant interface vnet0.IPv6 for mDNS. Dec 01 14:35:33 linuxServer avahi-daemon[1646]: Registering new address record for fe80::fc54:ff:fe92:d47b on vnet0.*. Dec 01 14:35:34 linuxServer kernel: virbr0: port 2(vnet0) entered learning state Dec 01 14:35:35 linuxServer kernel: vfio-pci 0000:09:00.0: vfio_ecap_init: hiding ecap 0x1e@0x178 Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) config/udev: removing device Logitech M215 2nd Gen Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (**) Option "fd" "75" Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) event2 - Logitech M215 2nd Gen: device removed Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) UnloadModule: "libinput" Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) systemd-logind: releasing fd for 13:66 Dec 01 14:35:35 linuxServer acpid[1645]: input device has been disconnected, fd 21 Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) config/udev: removing device Logitech K330 Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (**) Option "fd" "77" Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) UnloadModule: "libinput" Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) systemd-logind: not releasing fd for 13:67, still in use Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) config/udev: removing device Logitech K330 Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (**) Option "fd" "77" Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) event3 - Logitech K330: device removed Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) UnloadModule: "libinput" Dec 01 14:35:35 linuxServer /usr/libexec/gdm-x-session[2634000]: (II) systemd-logind: releasing fd for 13:67 Dec 01 14:35:35 linuxServer kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x270 Dec 01 14:35:35 linuxServer kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0 Dec 01 14:35:35 linuxServer kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x370 Dec 01 14:35:36 linuxServer kernel: vfio-pci 0000:05:00.0: vfio_ecap_init: hiding ecap 0x1e@0x154 Dec 01 14:35:36 linuxServer NetworkManager[1648]: [1606854936.8911] device (virbr0): carrier: link connected Dec 01 14:35:36 linuxServer kernel: virbr0: port 2(vnet0) entered forwarding state Dec 01 14:35:36 linuxServer kernel: virbr0: topology change detected, propagating Dec 01 14:35:42 linuxServer systemd-resolved[1517]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Dec 01 14:35:42 linuxServer systemd-resolved[1517]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Dec 01 14:35:42 linuxServer systemd-resolved[1517]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Dec 01 14:35:42 linuxServer systemd-resolved[1517]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Dec 01 14:35:42 linuxServer systemd-resolved[1517]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP. Dec 01 14:35:42 linuxServer systemd[1]: NetworkManager-dispatcher.service: Succeeded. Step #3, no working USB keyboard/mouse above, so virsh reset macOS, then working USB keyboard/mouse, and, a) lsusb -t /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M |__ Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M |__ Port 3: Dev 26, If 0, Class=Human Interface Device, Driver=usbfs, 1.5M |__ Port 4: Dev 3, If 0, Class=Communications, Driver=rndis_host, 480M |__ Port 4: Dev 3, If 1, Class=CDC Data, Driver=rndis_host, 480M /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M |__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=uas, 10000M /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M |__ Port 1: Dev 4, If 2, Class=Human Interface Device, Driver=usbfs, 12M |__ Port 1: Dev 4, If 0, Class=Human Interface Device, Driver=usbfs, 12M |__ Port 1: Dev 4, If 1, Class=Human Interface Device, Driver=usbfs, 12M |__ Port 7: Dev 3, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M |__ Port 10: Dev 5, If 0, Class=Vendor Specific Class, Driver=pl2303, 12M b) Output from, journalctl -f | grep -v x11vnc, Dec 01 14:36:10 linuxServer kernel: usb 1-1.1: reset full-speed USB device number 4 using xhci_hcd Dec 01 14:36:10 linuxServer upowerd[4208]: treating change event as add on /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.0/usb1/1-1/1-1.1 Dec 01 14:36:10 linuxServer upowerd[4208]: treating change event as add on /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.0/usb1/1-1/1-1.1 Make sense? Thanks! The logs make total sense, thank you! OK on lsusb: We see the HID devices flip from "usbhid" driver to "usbfs" with the initial start. They no more change on the reset. On the journal entries we see on the initial guest start that gnome has to yield the devices (OK). USB isn't mentioned at all on this initial start No errors that would seem obvious there. ... Then on the reset we see (due to the reset) that the host kernel seems to do a USB reset Dec 01 14:36:10 linuxServer kernel: usb 1-1.1: reset full-speed USB device number 4 using xhci_hcd Dec 01 14:36:10 linuxServer upowerd[4208]: treating change event as add on /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.0/usb1/1-1/1-1.1 Dec 01 14:36:10 linuxServer upowerd[4208]: treating change event as add on /sys/devices/pci0000:00/0000:00:01.3/0000:02:00.0/usb1/1-1/1-1.1 That does not immediately point to the issue unfortunately :-/ I assume the bus reset is done as part of giving the devices back and resetting them. Maybe we should try to reset this device when the hang happens ourselve to see if it does something: Be warned, this could be bad as in "need to reboot afterwards" Device: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 480M I'm unsure which reset exactly that was, maybe the following (re-)initializes too much but you might try (when the guest hangs): $ cd /sys/bus/pci/drivers/xhci_hcd/; echo -n "0000:00:01.3" > unbind; echo -n "0000:00:01.3" > bind; On a similar note, any success if you replug one of those devices? --- On (2) you said: "I can't say if this happens with other guests or not, as I only bind this USB device to this particular (guest) OS" => But you can for a try shut that guest down and use another e.g. Ubuntu guest with the same forwarding to try right? No issue with reboot, worry about that after we debug this :-). I tried the noted command, but I get, bash: echo: write error: No such device bash: echo: write error: No such device FYI, $ ls -alF /sys/bus/pci/drivers/xhci_hcd/ total 0 drwxr-xr-x 2 root root 0 Dec 2 17:51 ./ drwxr-xr-x 35 root root 0 Dec 2 17:51 ../ lrwxrwxrwx 1 root root 0 Dec 2 17:51 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.3/0000:02:00.0/ lrwxrwxrwx 1 root root 0 Dec 2 17:51 0000:0b:00.3 -> ../../../../devices/pci0000:00/0000:00:07.1/0000:0b:00.3/ --w------- 1 root root 4096 Dec 2 17:53 bind lrwxrwxrwx 1 root root 0 Dec 2 17:51 module -> ../../../../module/xhci_pci/ --w------- 1 root root 4096 Dec 2 17:51 new_id --w------- 1 root root 4096 Dec 2 17:51 remove_id --w------- 1 root root 4096 Dec 2 17:51 uevent --w------- 1 root root 4096 Dec 2 17:53 unbind So I tried echo -n "0000:02:00.0", but no recovery. Also, unplug / plug ... nope. So then I tried my old standby :-). virsh reset macOS => Nope, not working now. Had to destroy the VM, then the usual (start, reset) ... working again. Thoughts? Also just stumbled on to this by accident ... on VM start, no USB (as above). But leave it 5 min, and the VM reboots itself (internally, not restart at the virsh "level" ... UEFI watchdog perhaps?). And then USB is OK. So is it some sort of race condition / timing issue at initial VM startup? Thanks! Hmm, I was guessing the paths from your former log output. Thanks for fixing it up as needed on your system. So we know that just resetting it from the Host POV won't change anything. And thanks for the info on "auto-recovering" on guest reboot. The one obvious next test would be to shut down the macOS guest and try an ubuntu guest with the same forwards. That will allow us to see if it is "virtualization-components" or actually "guest OS" that we need to think about. > The one obvious next test would be to shut down the macOS guest and try an ubuntu guest with the same forwards. That will allow us to see if it is "virtualization-components" or actually "guest OS" that we need to think about. Good idea. OK, so I shut down all VM's, added this same USB device to a Windows 10 VM I also had defined (on the same host OS). Started it up (same way, virsh start) => booted up, and ... USB device works! So it seems to be guest OS related, agreed? And actually, not even that I'd say, as I can't even use it in the bootloader / UEFI, so not really even the OS, but the bootloader / UEFI? And that sort of aligns with the watchdog reset, as then it works, but still hasn't gotten to the OS (boot). Agreed? Thanks! I think we can agree that it is "guest dependent" Either Windows has a way to fix up the issue or it doesn't even have it to begin with (compare to MacOS). Chances are that your (virtual) bootloader/UEFI are what breaks it to begin with (and not the MacOS later). Do you think you could play around with the options you have for that (does it need to be UEFI for example)? Even if you'd need to stick to UEFI you could reset it by e.g. replacing it's VAR file that represents the UEFI-writable space. OK, some interesting findings :-). And back to thinking it may be QEMU related? Let me explain, by all means disagree! 1) I found that using OpenCore with macOS, I no longer need a custom UEFI. So, I tried the UEFI that comes with Ubuntu (apt install ovmf, copied the files over). No change. So then, I cloned and built the latest UEFI from Tianocore / EDK II (https://github.com/tianocore/edk2). Copied that over (and VARS, default), no delta. And as I can't get into the UEFI menu on VM startup (i.e. press ESC, F2, etc.) ... seems to be OS independent, just an issue between UEFI and QEMU - agreed? 2) Figured I'd try to clone, build and run the latest version of QEMU, see if that is it - FYI, the current version in Ubuntu that I'm running is, QEMU emulator version 5.0.0 (Debian 1:5.0-5ubuntu9.2) Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers OK, that said ... I can build, install => but can't get apparmor to let me run my custom QEMU executable (arrgh!). I even tried disabling security_driver (set to "none" in /etc/libvirt/qemu.conf, restarted libvirtd.service) ... but no joy. Can't get the latest QEMU to run. Any thoughts here would be appreciated! This is to see if it's an "older" issue, and may be resolved already. 3) An interesting side note, when I did get into the UEFI menu (post VM reset), changed the resolution ... I could no longer boot (fully) my guest OS. Had to reboot my host OS (Ubuntu), then all OK. Very odd, but an observation :-). Thanks for all the help! FYI, was able to get the following version of QEMU running (using the info at https://stackoverflow.com/questions/10817813/apparmor-causes-issues-on-libvirt-with-custom-qemu), QEMU emulator version 5.1.93 (v5.2.0-rc3) Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers No delta - still reacts the same. Thanks! Thanks for your feedback Russell. You resolved the apparmor issue faster than I could reply :-) About the actual issue, it indeed seems like an issue between UEFI and QEMU here. But I'm still wondering why it then would be any different with Windows and/or Linux guests. Is the UEFI in those Windows/Linux guests always working fine, or is in those cases maybe the UEFI affected but the later boot of the OS resets/fixes the bad condition (whatever it is). And if on those non MacOS guests the UEFI is always fine, then we are back wondering why that would be the case if they are using the same UEFI&Qemu combination. ... I feel like I can't really help here, but I'm happy to continue to discuss. Hopefully we can end in a place which either of us can actually action upon to resolve it for you. No worries, and thanks for all the help so far - it really is appreciated! On the apparmor issue, is this the "right" fix? I modified /etc/apparmor.d/abstractions/libvirt-qemu, adding, /usr/local/bin/qemu-system-x86_64 rmix, /usr/local/share/qemu/** r, Or is there a "better" (correct) way? As for Windows/Linux vs. macOS guests, I don't think they are usign the same UEFI (are they?). I did reboot the Windows guest, and it looks to be a different "BIOS", agreed? Checking the xml configuration files, they are a bit different (macOS copied below, Windows does not include this). So is it (specifically?) UEFI vs. QEMU (i.e. that is the issue)? OVMF_CODE.fd OVMF_VARS.fd I could definitely be wrong here, but thinking that "stock" UEFI should work / be supported? Thanks again! > On the apparmor issue, is this the "right" fix? I modified /etc/apparmor.d/abstractions/libvirt-qemu, adding, > /usr/local/bin/qemu-system-x86_64 rmix, > /usr/local/share/qemu/** r, > > Or is there a "better" (correct) way? Doing the same in /etc/apparmor.d/local/abstractions/libvirt-qemu is better to now get conflicts on upgrades. > As for Windows/Linux vs. macOS guests, I don't think they are usign the same UEFI (are they?). > I did reboot the Windows guest, and it looks to be a different "BIOS", agreed? > Checking the xml configuration files, they are a bit different (macOS copied below, Windows does not include this). So is it (specifically?) UEFI vs. QEMU (i.e. that is the issue)? > OVMF_CODE.fd > OVMF_VARS.fd The above section belongs to using OVMF and if the windows guest didn't have that it probably runs on seabios or some other non UEFI path. > Doing the same in /etc/apparmor.d/local/abstractions/libvirt-qemu is > better to now get conflicts on upgrades. Makes sense, thanks! Will move the items there. Both entries are needed? Seems like I can just restart apparmor after (or at least that's working for me). > if the windows guest didn't have that it probably runs on seabios or > some other non UEFI path. Agreed! Looks like it runs SeaBIOS (fast flash of the screen :-)). So the issue seems to be UEFI and QEMU? Thanks! On Mon, Dec 7, 2020 at 4:05 PM Russell Morris